Electronic device and method of executing function of electronic device

ABSTRACT

An electronic device and a method of the electronic device are provided. The electronic device includes a communication circuit, a microphone, a memory configured to store instructions, and at least one processor operatively connected with the communication circuit, the microphone, and the memory. Upon execution of the stored instructions, the at least one processor is configured to perform a first operation, receive an utterance through the microphone while performing the first operation, the utterance including at least one selected wake-up word used to invoke a speech-based intelligent service, determine, based on the utterance, a response related to the first operation, and output the response.

CROSS-REFERENCE TO RELATED APPLICATION(S)

This application is based on and claims priority under 35 U.S.C. §119(a) of a Korean patent application number 10-2018-0046141, filed onApr. 20, 2018, in the Korean Intellectual Property Office, thedisclosure of which is incorporated by reference herein in its entirety.

BACKGROUND 1. Field

The disclosure relates to electronic devices and methods of executingthe functions of the electronic devices.

2. Description of Related Art

Artificial intelligence (AI) systems are computer systems capable ofimplementing human-like intelligence which allows machines toself-learn, make decisions, and improve cognition as they are used moreand more.

AI technology may include element techniques, such as machine learning(deep learning), which utilize algorithms capable of classifying andlearning the features of entered data on their own and copying theperception or determination by the human brain using machine learningalgorithms.

Such element techniques may include linguistic understanding whichrecognizes human languages/words, visual understanding which recognizesthings as humans visually do, inference/prediction which judgesinformation and performs logical inference and prediction, knowledgeexpression which processes human experience information as knowledgedata, and motion control which controls robot motions and driverlessvehicles.

Linguistic understanding is technology for recognizing andapplying/processing a human being's language or text, and thisencompasses natural language processing, machine translation, dialogsystem, answering inquiries, and speech recognition/synthesis.

Inference prediction is a technique of determining, logically inferring,and predicting information, encompassing knowledge/probability-basedinference, optimization prediction, preference-based planning, andrecommendation.

A user may input his utterance to an electronic device operating in anAI system. When the electronic device is a predetermined distance awayfrom the user or is operating in a screen lock (hold) state, the usermay have difficulty in identifying whether the operation correspondingto the utterance is being performed.

The above information is presented as background information only toassist with an understanding of the disclosure. No determination hasbeen made, and no assertion is made, as to whether any of the abovemight be applicable as prior art with regard to the disclosure.

SUMMARY

Aspects of the disclosure are to address at least the above-mentionedproblems and/or disadvantages and to provide at least the advantagesdescribed below. Accordingly, an aspect of the disclosure is to providean electronic device and method of executing a function of theelectronic device for activating an intelligence system when receivingan input of an utterance (e.g., a wake-up utterance) constituted ofdesignated words and output information about an application running onthe electronic device as feedback for the utterance.

Additional aspects will be set forth in part in the description whichfollows and, in part, will be apparent from the description, or may belearned by practice of the presented embodiments.

In accordance with an aspect of the disclosure, an electronic device isprovided. The electronic device includes a communication circuit, amicrophone, a memory configured to store instructions, and at least oneprocessor operatively connected with the communication circuit, themicrophone, and the memory. Upon execution of the stored instructions,the at least one processor configured to perform a first operation,receive an utterance through the microphone while performing the firstoperation, the utterance including at least one selected wake-up wordused to invoke a speech-based intelligent service, determine, based onthe utterance, a response related to the first operation, and output theresponse.

In accordance with another aspect of the disclosure, a method ofexecuting a function of an electronic device is provided. The methodincludes performing a first operation, receiving an utterance through amicrophone of the electronic device while performing the firstoperation, determining, based on the utterance, a response related tothe first operation, and outputting the response. The utterance mayinclude only at least one selected wake-up word used to invoke aspeech-based intelligent service.

Other aspects, advantages, and salient features of the disclosure willbecome apparent to those skilled in the art from the following detaileddescription, which, taken in conjunction with the annexed drawings,discloses various embodiments of the disclosure.

BRIEF DESCRIPTION OF THE DRAWINGS

The above and other aspects, features, and advantages of certainembodiments of the disclosure will be more apparent from the followingdescription taken in conjunction with the accompanying drawings, inwhich:

FIG. 1 is a view illustrating an integrated intelligence systemaccording to an embodiment of the disclosure;

FIG. 2 is a block diagram illustrating a user terminal in an integratedintelligence system according to an embodiment of the disclosure;

FIG. 3 is a view illustrating an example of executing an intelligent appon a user terminal according to an embodiment of the disclosure;

FIG. 4 is a block diagram illustrating an intelligent server in anintegrated intelligence system according to an embodiment of thedisclosure;

FIG. 5 is a view illustrating an example of identifying current statesby a context module of an intelligent service module according to anembodiment of the disclosure;

FIG. 6 is a block diagram illustrating a proposal module of anintelligent service module according to an embodiment of the disclosure;

FIG. 7 is a view illustrating a method for generating a path rule by apath planner module according to an embodiment of the disclosure;

FIG. 8 is a view illustrating an example of managing user information bya persona module of an intelligent service module according to anembodiment of the disclosure;

FIG. 9 is a flowchart illustrating an example operation of executing afunction of an electronic device in an intelligence system according toan embodiment of the disclosure;

FIG. 10 is a flowchart illustrating an example operation of executing afunction of an electronic device in an intelligence system according toan embodiment of the disclosure;

FIG. 11 is a flowchart illustrating an example operation between aserver and an electronic device in an intelligence system according toan embodiment of the disclosure;

FIG. 12 is a block diagram illustrating example components of anintelligence system and example operations among the componentsaccording to an embodiment of the disclosure;

FIG. 13 is a flowchart illustrating an example operation between aserver and an electronic device in an intelligence system according toan embodiment of the disclosure;

FIG. 14 is a flowchart illustrating an example operation between aserver and an electronic device in an intelligence system according toan embodiment of the disclosure;

FIG. 15 is a block diagram illustrating example components of anintelligence system and example operations among the componentsaccording to an embodiment of the disclosure;

FIG. 16 is a view illustrating an example screen configuration ofsetting an application to use an intelligent service according to anembodiment of the disclosure;

FIGS. 17A, 17B, and 17C are views illustrating various example scenariosof outputting feedback by an electronic device according to entry of auser's utterance in an intelligence system according to an embodiment ofthe disclosure;

FIG. 18 is a view illustrating an example scenario of outputtingfeedback from a smart speaker according to entry of a user's utterancein an intelligence system according to an embodiment of the disclosure;and

FIG. 19 is a block diagram illustrating an electronic device in anetwork environment according to an embodiment of the disclosure.

Throughout the drawings, like reference numerals will be understood torefer to like parts, components, and structures.

DETAILED DESCRIPTION

The following description with reference to the accompanying drawings isprovided to assist in a comprehensive understanding of variousembodiments of the disclosure as defined by the claims and theirequivalents. It includes various specific details to assist in thatunderstanding but these are to be regarded as merely exemplary.Accordingly, those of ordinary skill in the art will recognize thatvarious changes and modifications of the various embodiments describedherein can be made without departing from the scope and spirit of thedisclosure. In addition, descriptions of well-known functions andconstructions may be omitted for clarity and conciseness.

The terms and words used in the following description and claims are notlimited to the bibliographical meanings, but, are merely used by theinventor to enable a clear and consistent understanding of thedisclosure. Accordingly, it should be apparent to those skilled in theart that the following description of various embodiments of thedisclosure is provided for illustration purpose only and not for thepurpose of limiting the disclosure as defined by the appended claims andtheir equivalents.

It is to be understood that the singular forms “a,” “an,” and “the”include plural references unless the context clearly dictates otherwise.Thus, for example, reference to “a component surface” includes referenceto one or more of such surfaces.

As used herein, the terms “A or B” or “at least one of A and/or B” mayinclude all possible combinations of A and B. As used herein, the terms“first” and “second” may modify various components regardless ofimportance and/or order and are used to distinguish a component fromanother without limiting the components. It will be understood that whenan element (e.g., a first element) is referred to as being (operativelyor communicatively) “coupled with/to,” or “connected with/to” anotherelement (e.g., a second element), it can be coupled or connected with/tothe other element directly or via a third element.

As used herein, the terms “configured to” may be interchangeably usedwith other terms, such as “suitable for,” “capable of,” “modified to,”“made to,” “adapted to,” “able to,” or “designed to” in hardware orsoftware in the context. Rather, the term “configured to” may mean thata device can perform an operation together with another device or parts.For example, the term “processor configured (or set) to perform A, B,and C” may mean a generic-purpose processor (e.g., a central processingunit (CPU) or an application processor (AP)) that may perform theoperations by executing one or more software programs stored in a memorydevice or a dedicated processor (e.g., an embedded processor) forperforming the operations.

For example, examples of the electronic device according to embodimentsof the disclosure may include at least one of a smartphone, a tabletpersonal computer (PC), a mobile phone, a video phone, an e-book reader,a desktop PC, a laptop computer, a netbook computer, a workstation, aserver, a personal digital assistant (PDA), a portable multimedia player(PMP), a Moving Picture Experts Group phase 1 or phase 2 (MPEG-1 orMPEG-2) audio layer 3 (MP3) player, a medical device, a camera, or awearable device. The wearable device may include at least one of anaccessory-type device (e.g., a watch, a ring, a bracelet, an anklet, anecklace, glasses, contact lenses, or a head-mounted device (HMD)), afabric- or clothes-integrated device (e.g., electronic clothes), a bodyattaching-type device (e.g., a skin pad or tattoo), or a bodyimplantable device. In some embodiments, examples of the smart homeappliance may include at least one of a television (TV), a digitalversatile disc (DVD) player, an audio player, a refrigerator, an airconditioner, a cleaner, an oven, a microwave oven, a washer, a drier, anair cleaner, a set-top box, a home automation control panel, a securitycontrol panel, a TV box (e.g., Samsung HomeSync™, Apple TV™, or GoogleTV™), a gaming console (Xbox™, PlayStation™), an electronic dictionary,an electronic key, a camcorder, or an electronic picture frame.

According to an embodiment of the disclosure, the electronic device mayinclude at least one of various medical devices (e.g., diverse portablemedical measuring devices (a blood sugar measuring device, a heartbeatmeasuring device, or a body temperature measuring device), a magneticresource angiography (MRA) device, a magnetic resource imaging (MRI)device, a computed tomography (CT) device, an imaging device, or anultrasonic device), a navigation device, a global navigation satellitesystem (GNSS) receiver, an event data recorder (EDR), a flight datarecorder (FDR), an automotive infotainment device, a sailing electronicdevice (e.g., a sailing navigation device or a gyro compass), avionics,security devices, vehicular head units, industrial or home robots,drones, automatic teller machines (ATMs), point of sale (POS) devices,or internet of things (IoT) devices (e.g., a bulb, various sensors, asprinkler, a fire alarm, a thermostat, a street light, a toaster,fitness equipment, a hot water tank, a heater, or a boiler). Accordingto various embodiments of the disclosure, examples of the electronicdevice may at least one of part of a piece of furniture,building/structure or vehicle, an electronic board, an electronicsignature receiving device, a projector, or various measurement devices(e.g., devices for measuring water, electricity, gas, or electromagneticwaves). According to embodiments of the disclosure, the electronicdevice may be flexible or may be a combination of the above-enumeratedelectronic devices.

According to an embodiment of the disclosure, the electronic devices arenot limited to those described above. As used herein, the term “user”may denote a human or another device (e.g., an artificially intelligentelectronic device) using the electronic device.

FIG. 1 is a view illustrating an integrated intelligence systemaccording to an embodiment of the disclosure.

Referring to FIG. 1, an integrated intelligence system 10 may include auser terminal 100, an intelligent server 200, a personal informationserver 300, or a proposal server 400.

The user terminal 100 may provide services necessary for the userthrough an app (or application program) (e.g., an alarm app, messageapp, photo (gallery) app, etc.) stored in the user terminal 100. Forexample, the user terminal 100 may execute and operate another appthrough an intelligent app (or speech recognition app) stored in theuser terminal 100. The intelligent app of the user terminal 100 mayreceive user inputs to execute and operate the other app through theintelligent app. The user inputs may be received through, e.g., aphysical button, touchpad, speech input, or remote input. According toan embodiment of the disclosure, the user terminal 100 may be variousterminal devices (or electronic devices) connectable to the internet,such as a cellular phone, smartphone, PDA, or laptop computer.

According to an embodiment of the disclosure, the user terminal 100 mayreceive an utterance as a user input. The user terminal 100 may receivethe utterance and generate a command to operate the app based on theutterance. Accordingly, the user terminal 100 may operate the app usingthe command.

The intelligent server 200 may receive the user's speech input from theuser terminal 100 through a communication network and convert the speechinput into text data. According to an embodiment of the disclosure, theintelligent server 200 may generate (or select) a path rule based on thetext data. The path rule may include information about actions (oroperations) to perform the functions of the app or information aboutparameters necessary to execute the operations. Further, the path rulemay include the order of the operations of the app. The user terminal100 may receive the path rule, select an app according to the path rule,and execute the operations included in the path rule on the selectedapp.

As used herein, the term “path rule” may generally mean a sequence ofstates for the electronic device to perform a task requested by theuser, but not limited thereto. In other words, the path rule may containinformation about a sequence. The task may be a certain action that,e.g., an intelligent app may provide. The task may include producing aschedule, transmitting a photo to a desired party, or providing weatherinformation. The user terminal 100 may perform the task by sequentiallyhaving at least one or more states (e.g., operation states of the userterminal 100).

According to an embodiment, the path rule may be provided or created byan artificial intelligence (AI) system. The AI system may be arule-based system or a neural network-based system (e.g., feedforwardneural network (FNN)) or recurrent neutral network (RNN)). Or, the AIsystem may be a combination thereof or a system different therefrom.According to an embodiment, the path rule may be selected from a set ofpre-defined path rules or created in real-time in response to a userrequest. For example, the AI system may select at least one among aplurality of pre-defined path rules or may dynamically (or real-time)create a path rule. The user terminal 100 may use a hybrid system toprovide the path rule.

According to an embodiment, the user terminal 100 may execute theoperation and display, on the display, the screen corresponding to thestate of the user terminal 100 having performed the operation. Asanother example, the user terminal 100 may execute the operation andabstain from displaying the results of performing the operation on thedisplay. The user terminal 100 may execute, e.g., a plurality ofoperations and display, on the display, only some results of theplurality of operations. The user terminal 100 may display, on thedisplay, e.g., the results of executing only the last operation inorder. As another example, the user terminal 100 may receive a userinput and display the results of executing the operation on the display.

The personal information server 300 may include a database storing userinformation. For example, the personal information server 300 mayreceive user information (e.g., context information or app execution)from the user terminal 100 and store the user information in thedatabase. The intelligent server 200 may receive the user informationfrom the personal information server 300 through the communicationnetwork and use the same in creating a path rule for user inputs.According to an embodiment of the disclosure, the user terminal 100 mayreceive user information from the personal information server 300through the communication network and use the same as information formanaging the database.

The proposal server 400 may include a database that stores informationabout functions to be provided or introductions of applications orfunctions in the terminal. For example, the proposal server 400 mayreceive user information of the user terminal 100 from the personalinformation server 300 and include a database for functions that theuser may use. The user terminal 100 may receive information aboutfunctions to be provided from the proposal server 400 through thecommunication network and provide the information to the user.

FIG. 2 is a block diagram illustrating a user terminal in an integratedintelligence system according to an embodiment of the disclosure.

Referring to FIG. 2, the user terminal 100 may include an input module110, a display 120, a speaker 130, a memory 140, and a processor 150.The user terminal 100 may further include a housing. The components ofthe user terminal 100 may be positioned in or on the housing. The userterminal 100 may further include a communication circuit positionedinside the housing. The user terminal 100 may transmit or receive data(or information) to/from an external server (e.g., the intelligentserver 200) through the communication circuit.

According to an embodiment of the disclosure, the input module 110 mayreceive user inputs from the user. For example, the input module 110 mayreceive a user input from an external device (e.g., a keyboard orheadset) connected thereto. As another example, the input module 110 mayinclude a touchscreen combined with the display 120 (e.g., a touchscreendisplay). As another example, the input module 110 may include ahardware key (or a physical key) 112 positioned in the user terminal 100(or the housing of the user terminal 100).

According to an embodiment, the input module 110 may include amicrophone capable of receiving utterances as speech signals. Forexample, the input module 110 may include a speech input system andreceive utterances as speech signals through the speech input system.The microphone may be exposed through, e.g., a portion (e.g., a firstportion) of the housing.

According to an embodiment of the disclosure, the display 120 maydisplay images, videos, and/or application execution screens. Forexample, the display 120 may display a graphical user interface (GUI) ofthe app. According to an embodiment, the display 120 may be exposedthrough, e.g., a portion (e.g., a second portion) of the housing.

According to an embodiment of the disclosure, the speaker 130 may outputspeech signals. For example, the speaker 130 may output speech signalsgenerated from inside the user terminal 100 to the outside. According toan embodiment, the speaker 130 may be exposed through, e.g., a portion(e.g., a third portion) of the housing.

According to an embodiment of the disclosure, the memory 140 may store aplurality of apps 141 and 143. The plurality of apps 141 and 143 may be,e.g., programs for performing a function corresponding to a user input.According to an embodiment of the disclosure, the memory 140 may includethe intelligent agent 145, the execution manager module 147, or theintelligent service module 149. The intelligent agent 145, the executionmanager module 147, and the intelligent service module 149 may beframeworks (or application frameworks) to process received user inputs(e.g., utterances).

According to an embodiment of the disclosure, the memory 140 may includea database that may store information necessary to recognize userinputs. For example, the memory 140 may include a log database capableof storing log information. As another example, the memory 140 mayinclude a persona database capable of storing user information.

According to an embodiment of the disclosure, the memory 140 may storethe plurality of apps 141 and 143. The plurality of apps 141 and 143 maybe loaded and operated. For example, the plurality of apps 141 and 143stored in the memory 140 may be loaded and operated by the executionmanager module 147. The plurality of apps 141 and 143 may includeexecution service modules 141 a and 143 a to perform functions.According to an embodiment, the plurality of apps 141 and 143 mayexecute a plurality of operations (e.g., a sequence of states) 141 b and143 b through the execution service modules 141 a and 143 a to performfunctions. In other words, the execution service modules 141 a and 143 amay be activated by the execution manager module 147 and execute theplurality of operations 141 b and 143 b.

According to an embodiment of the disclosure, when the operations 141 band 143 b of the apps 141 and 143 are executed, the execution statescreens as per the execution of the operations 141 b and 143 b may bedisplayed on the display 120. The execution state screens may bescreens, e.g., in the state of the operations 141 b and 143 b havingbeen completed. The execution state screens may be screens, e.g., in thestate of the execution of the operations 141 b and 143 b having beenstopped (partial landing) (e.g., when parameters required for theoperations 141 b and 143 b are not inputted).

According to an embodiment of the disclosure, the execution servicemodules 141 a and 143 a may execute the operations 141 b and 143 b asper a path rule. For example, the execution service modules 141 a and143 a may be activated by the execution manager module 147, receive anexecution request as per the path rule from the execution manager module147, and execute the operations 141 b and 143 b according to theexecution request, thereby executing the functions of the apps 141 and143. The execution service modules 141 a and 143 a may send completioninformation to the execution manager module 147 when the execution ofthe operations 141 b and 143 b is complete.

According to an embodiment of the disclosure, when the plurality ofoperations 141 b and 143 b are executed on the apps 141 and 143, theplurality of operations 141 b and 143 b may sequentially be executed.When the execution of one operation (e.g., operation 1 of the first app141 or operation 1 of the second app 143) is complete, the executionservice modules 141 a and 143 a may open the subsequent operation (e.g.,operation 2 of the first app 141 or operation 2 of the second app 143)and send completion information to the execution manager module 147.Here, ‘open an operation’ may be appreciated as transitioning theoperation into an executable state or preparing for the execution of theoperation. In other words, unless the operation is open, the operationcannot be executed. Upon receiving the completion information, theexecution manager module 147 may transfer an execution request for thesubsequent operation (e.g., operation 2 of the first app 141 oroperation 2 of the second app 143) to the execution service module.According to an embodiment of the disclosure, when the plurality of apps141 and 143 are executed, the plurality of apps 141 and 143 maysequentially be executed. For example, when the execution of the lastoperation of the first app 141 (e.g., operation 3 of the first app 141)is complete, and completion information is thus received, the executionmanager module 147 may send an execution request for the first operationof the second app 143 (e.g., operation 1 of the second app 143) to theexecution service module 143 a.

According to an embodiment of the disclosure, when the plurality ofoperations 141 b and 143 b are executed on the apps 141 and 143, theresultant screens of execution of the plurality of operations 141 b and143 b may be displayed on the display 120. According to an embodiment ofthe disclosure, only some of the plurality of resultant screens ofexecution of the plurality of operations 141 b and 143 b may bedisplayed on the display 120.

According to an embodiment of the disclosure, the memory 140 may storean intelligent app (e.g., a speech recognition app) interworking withthe intelligent agent 145. The app interworking with the intelligentagent 145 may receive an utterance as a speech signal and process thesame. According to an embodiment of the disclosure, the app interworkingwith the intelligent agent 145 may be operated by particular inputsentered through the input module 110 (e.g., inputs through the hardwarekey or touchscreen, or particular speech inputs).

According to an embodiment of the disclosure, the intelligent agent 145,the execution manager module 147, or the intelligent service module 149stored in the memory 140 may be executed by the processor 150. Thefunctions of the intelligent agent 145, the execution manager module147, or the intelligent service module 149 may be implemented by theprocessor 150. The functions of the intelligent agent 145, the executionmanager module 147, and the intelligent service module 149 are describedin connection with operations of the processor 150. According to anembodiment of the disclosure, the intelligent agent 145, the executionmanager module 147, or the intelligent service module 149 stored in thememory 140 may be implemented in software or hardware.

According to an embodiment of the disclosure, the processor 150 maycontrol the overall operation of the user terminal 100. For example, theprocessor 150 may control the input module 110 to receive user inputs.The processor 150 may control the display 120 to display images. Theprocessor 150 may control the speaker 130 to output speech signals. Theprocessor 150 may control the memory 140 to execute programs and fetchor store necessary information.

According to an embodiment of the disclosure, the processor 150 mayexecute the intelligent agent 145, the execution manager module 147, orthe intelligent service module 149 stored in the memory 140. Thus, theprocessor 150 may implement the function of the intelligent agent 145,the execution manager module 147, or the intelligent service module 149.

According to an embodiment of the disclosure, the processor 150 mayexecute the intelligent agent 145 to generate commands to operate appsbased on speech signals received as user inputs. According to anembodiment of the disclosure, the processor 150 may execute theexecution manager module 147 to execute the apps 141 and 143 stored inthe memory 140 as per the commands generated. According to anembodiment, the processor 150 may execute the intelligent service module149 to manage the user information and process user inputs using theuser information.

The processor 150 may execute the intelligent agent 145 to send userinputs received through the input module 110 to the intelligent server200 and process the user inputs through the intelligent server 200.

According to an embodiment of the disclosure, the processor 150 mayexecute the intelligent agent 145 to pre-process the user inputs beforesending the user inputs to the intelligent server 200. According to anembodiment of the disclosure, the intelligent agent 145 may include anadaptive echo canceller (AEC) module, a noise suppression (NS) module,an end-point detection (EPD) module, or an automatic gain control (AGC)module to pre-process the user inputs. The AEC module may remove echoesmixed in the user inputs. The NS module may suppress background noisemixed in the user inputs. The EPD module may detect end points of userspeech contained in the user inputs to find where the user speech ispresent using the detected end points. The AGC module may recognize theuser inputs and adjust the volume of the user inputs for a properprocessing of the recognized user inputs. According to an embodiment ofthe disclosure, although able to execute all of the pre-processingcomponents described above to provide a better performance, theprocessor 150 may alternatively execute some of the pre-processingcomponents to be operated at reduced power.

According to an embodiment of the disclosure, the intelligent agent 145may include a wake-up recognition module stored in the memory 140 torecognize the invocation. Thus, the processor 150 may recognize thewake-up command through the wake-up recognition module, and uponreceiving the wake-up command, the processor 150 may execute theintelligent agent 145 to receive user inputs. The wake-up recognitionmodule may be implemented in a low-power processor (e.g., a processorincluded in an audio codec). According to an embodiment, the processor150 may execute the intelligent agent 145 upon receiving a user inputthrough the hardware key. When the intelligent agent 145 is executed, anintelligent app (e.g., a speech recognition app) interworking with theintelligent agent 145 may be executed.

According to an embodiment of the disclosure, the intelligent agent 145may include a speech recognition module to execute user inputs. Theprocessor 150 may receive user inputs to execute operations on the appthrough the speech recognition module. For example, the processor 150may recognize, through the speech recognition module, limited user(speech) inputs (e.g., the “click” sound made when the capturingoperation is executed on the camera app) for executing operations, suchas the wake-up command on the apps 141 and 143. The processor 150 mayassist the intelligent server 200 to recognize and quickly process usercommands, which are processable in the user terminal 100, through thespeech recognition module. According to an embodiment, the speechrecognition module of the intelligent agent 145 to execute user inputsmay be implemented in an app processor.

According to an embodiment of the disclosure, the speech recognitionmodule (including the speech recognition module of the wake-uprecognition module) of the intelligent agent 145 may recognize userinputs using an algorithm for recognizing speech. The algorithm used torecognize speech may be at least one of, e.g., a hidden Markov model(HMM) algorithm, an artificial neural network (ANN) algorithm, or adynamic time warping (DTW) algorithm.

According to an embodiment of the disclosure, the processor 150 mayexecute the intelligent agent 145 to convert the speech inputs into textdata. For example, the processor 150 may send a user speech through theintelligent agent 145 to the intelligent server 200 and receive textdata corresponding to the user speech from the intelligent server 200.Thus, the processor 150 may display the converted text data on thedisplay 120.

According to an embodiment of the disclosure, the processor 150 mayexecute the intelligent agent 145 to receive a path rule from theintelligent server 200. According to an embodiment, the processor 150may transfer the path rule to the execution manager module 147 throughthe intelligent agent 145.

According to an embodiment of the disclosure, the processor 150 mayexecute the intelligent agent 145 to transfer an execution result log asper the path rule received from the intelligent server 200 to theintelligent service module 149. The execution result log transferred maybe accrued and managed in user preference information of a personamodule 149 b.

According to an embodiment of the disclosure, the processor 150 mayexecute the execution manager module 147 to receive the path rule fromthe intelligent agent 145, execute the apps 141 and 143, and allow theapps 141 and 143 to execute the operations 141 b and 143 b contained inthe path rule. For example, the processor 150 may send commandinformation (e.g., path rule information) to execute the operations 141b and 143 b to the apps 141 and 143 through the execution manager module147 and receive completion information about the operations 141 b and143 b from the apps 141 and 143.

According to an embodiment of the disclosure, the processor 150 mayexecute the execution manager module 147 to transfer command information(e.g., path rule information) to execute the operations 141 b and 143 bof the apps 141 and 143 between the intelligent agent 145 and the apps141 and 143. The processor 150 may bind the apps 141 and 143 to beexecuted as per the path rule through the execution manager module 147and transfer the command information (e.g., path rule information) aboutthe operations 141 b and 143 b contained in the path rule to the apps141 and 143. For example, the processor 150 may sequentially transferthe operations 141 b and 143 b contained in the path rule to the apps141 and 143 through the execution manager module 147, sequentiallyexecuting the operations 141 b and 143 b of the apps 141 and 143 as perthe path rule.

According to an embodiment, the processor 150 may execute the executionmanager module 147 to manage the execution states of the operations 141b and 143 b of the apps 141 and 143. For example, the processor 150 mayreceive information about the execution states of the operations 141 band 143 b from the apps 141 and 143 through the execution manager module147. When the execution states of the operations 141 b and 143 b are,e.g., partial landing states (e.g., when no parameters required for theoperations 141 b and 143 b are entered), the processor 150 may transferinformation about the partial landing states to the intelligent agent145 through the execution manager module 147. The processor 150 mayrequest the user to enter necessary information (e.g., parameterinformation) using the information transferred through the intelligentagent 145. When the execution states of the operations 141 b and 143 bare other states, e.g., operation states, the processor 150 may receivean utterance from the user through the intelligent agent 145. Theprocessor 150 may transfer information about the apps 141 and 143 beingexecuted through the execution manager module 147 and the executionstates of the apps 141 and 143 to the intelligent agent 145. Theprocessor 150 may send the utterance through the intelligent agent 145to the intelligent server 200. The processor 150 may receive parameterinformation about the utterance from the intelligent server 200 throughthe intelligent agent 145. The processor 150 may transfer the parameterinformation received through the intelligent agent 145 to the executionmanager module 147. The execution manager module 147 may change theparameters of the operations 141 b and 143 b into new parameters usingthe received parameter information.

According to an embodiment, the processor 150 may execute the executionmanager module 147 to transfer the parameter information contained inthe path rule to the apps 141 and 143. When the plurality of apps 141and 143 are sequentially executed as per the path rule, the executionmanager module 147 may deliver the parameter information contained inthe path rule from one app to the other.

According to an embodiment of the disclosure, the processor 150 mayexecute the execution manager module 147 to receive a plurality of pathrules. The processor 150 may select a plurality of path rules based onthe utterance through the execution manager module 147. For example,when a utterance specifies a certain app 141 to execute some operation141 a but does not specify another app 143 to execute the otheroperation 143 b, the processor 150 may, through the execution managermodule 147, receive a plurality of different path rules by which thesame app 141 (e.g., a gallery app) to execute the operation 141 a isexecuted and a different app 143 (e.g., a message app or a telegram app)to execute the other operation 143 b is executed. The processor 150 mayexecute the same operations 141 b and 143 b (e.g., the same continuousoperations 141 b and 143 b) of the plurality of path rules through theexecution manager module 147. When the same operations have beenexecuted, the processor 150 may, through the execution manager module147, display, on the display 120, the state screen where the differentapps 141 and 143 each contained in a respective one of the plurality ofpath rules may be selected.

According to an embodiment of the disclosure, the intelligent servicemodule 149 may include a context module 149 a, a persona module 149 b,or a proposal module 149 c.

The processor 150 may execute the context module 149 a to identifycurrent states of the apps 141 and 143 from the apps 141 and 143. Forexample, the processor 150 may execute the context module 149 a toreceive context information indicating the current states of the apps141 and 143 to identify the current states of the apps 141 and 143through the received context information.

The processor 150 may execute the persona module 149 b to manage thepersonal information about the user using of the user terminal 100. Forexample, the processor 150 may execute the persona module 149 b toidentify use information and execution results of the user terminal 100and manage the user's personal information using the identified useinformation and execution results of the user terminal 100.

The processor 150 may execute the proposal module 149 c to predict theuser's intention and recommend a command for the user based on theuser's intention. For example, the processor 150 may execute theproposal module 149 c to recommend a command for the user given theuser's current state (e.g., time, place, context, or app).

According to an embodiment, a system 10 comprises a first electronicdevice (e.g., the user terminal 100) including a communication circuit,an input module 110 (e.g., a microphone and a speaker), at least oneprocessor (e.g., the processor 150) being part of the first electronicdevice or configured to remotely communicate with the first electronicdevice, and a memory 140 positioned on or outside the first electronicdevice and operatively connected with the at least one processor,wherein the memory may store instructions executed to enable the atleast one processor to configured to perform a first operation, receivea utterance through the microphone while performing the first operation,the utterance including only at least one selected wake-up word used toinvoke a speech-based intelligent service, identify a response relatedto the first operation, and provide the response through the speaker.

According to an embodiment, the first electronic device may include afirst processor among the at least one processor. The system may furthercomprise a server including a second processor among the at least oneprocessor.

According to an embodiment, the instructions may be configured to enablethe first processor to perform the first operation, after receiving theutterance, transmit data related to the first operation to the server,receive the response from the server, and output a speech responsethrough the speaker based on at least part of the response.

According to an embodiment, the instructions may be configured to enablethe second processor to receive data related to the utterance from thefirst electronic device, generate or select a response related to thefirst operation, and transmit the response to the first electronicdevice.

According to an embodiment, the first operation may include a callevent, an alarm event, a timer event, music play, video play, ahands-free event, or a content downloading event.

According to an embodiment, information included in the response mayinclude information about the first operation, information about aprogress of the first operation, or information about an operationsubsequent to the first operation.

According to an embodiment, the instructions may be configured to enablethe at least one processor to identify progress information varied oradded within a designated time after generating the response, andfurther output a first response including the identified progressinformation through the speaker.

According to an embodiment, the instructions may be configured to enablethe first processor to receive a user input for the response, identify asecond operation related to the first operation when the received userinput corresponds to a designated input, and further output a secondresponse to identify whether to perform the second operation.

According to an embodiment, the instructions may be configured to enablethe first processor to receive a user input for the response and whenthe received user input does not correspond to a designated input,perform at least one function corresponding to the first operation.

According to an embodiment, the instructions may be configured to enablethe first processor to transmit data related to the first operation tothe server when no additional utterance is received within a designatedtime after the utterance is received.

According to an embodiment, the instructions may be configured to enablethe at least one processor to identify whether the first electronicdevice is operated in a designated mode and, after receiving theutterance, transmit data related to the first operation to the server.For example, the designated mode may include a mode (eyes-free mode) inwhich the user terminal 100 recognizes the user's biometric informationand the user does not look at the user terminal 100 or is a designateddistance away or a hold mode in which the screen (e.g., the displaydevice 160) of the user terminal 100 is turned off as no user input isidentified for a designated time.

FIG. 3 is a view illustrating an example of executing an intelligent appon a user terminal according to an embodiment of the disclosure.

FIG. 3 illustrates an example in which the user terminal 100 receivesuser inputs and executes an intelligent app (e.g., a speech recognitionapp) interworking with the intelligent agent 145.

Referring to FIG. 3, according to an embodiment of the disclosure, theuser terminal 100 may execute an intelligent app to recognize speechthrough the hardware key 112. For example, when the user terminal 100receives user inputs through the hardware key 112, the user terminal 100may display a user interface (UI) 121 of the intelligent app on thedisplay 120. The user may touch a speech recognition button 121 a in theUI 121 of the intelligent app for speech entry 120 b with theintelligent app UI 121 displayed on the display 120. As another example,the user may continuously press the hardware key 112 for speech entry120 b.

According to an embodiment of the disclosure, the user terminal 100 mayexecute an intelligent app to recognize speech through the microphone111. For example, when a designated speech (e.g., “wake up!”) is entered(120 a) through the microphone 111, the user terminal 100 may displaythe intelligent app UI 121 on the display 120.

FIG. 4 is a block diagram illustrating an intelligent server in anintegrated intelligence system according to an embodiment of thedisclosure.

Referring to FIG. 4, an intelligent server 200 may include an automaticspeech recognition (ASR) module 210, a natural language understanding(NLU) module 220, a path planner module 230, a dialogue manager (DM)module 240, a natural language generator (NLG) module 250, or atext-to-speech (TTS) module 260. According to an embodiment, theintelligent server 200 may include a communication circuit, a memory,and a processor. The processor may execute commands stored in the memoryto drive the ASR module 210, the NLU module 220, the path planner module230, the DM module 240, the NLG module 250, and the TTS module 260. Theintelligent server 200 may transmit or receive data (or information)to/from an external electronic device (e.g., the user terminal 100)through the communication circuit.

The NLU module 220 or the path planner module 230 of the intelligentserver 200 may generate a path rule.

According to an embodiment of the disclosure, the ASR module 210 mayconvert user inputs received from the user terminal 100 into text data.

According to an embodiment of the disclosure, the ASR module 210 mayconvert user inputs received from the user terminal 100 into text data.For example, the ASR module 210 may include a speech recognition module.The speech recognition module may include an acoustic model and alanguage model. For example, the acoustic model may includevocalization-related information, and the language model may includeunit phonemic information and combinations of pieces of unit phonemicinformation. The speech recognition module may convert utterances intotext data using the vocalization-related information and unit phonemicinformation. Information about the acoustic model and the language modelmay be stored in, e.g., an ASR database (DB) 211.

According to an embodiment of the disclosure, the NLU module 220 mayperform syntactic analysis or semantic analysis to identify the user'sintent. As per the syntactic analysis, the user input may be dividedinto syntactic units (e.g., words, phrases, or morphemes) and whatsyntactic elements the syntactic units have may be identified. Thesemantic analysis may be performed using, e.g., semantic matching, rulematching, or formula matching. Thus, the NLU module 220 may obtain adomain, intent, or parameters (or slots) necessary to represent theintent for the user input.

According to an embodiment of the disclosure, the NLU module 220 maydetermine the user's intent and parameters using the matching rule whichhas been divided into the domain, intent, and parameters (or slots)necessary to identify the intent. For example, one domain (e.g., analarm) may include a plurality of intents (e.g., alarm settings orreleasing alarm), and one intent may include a plurality of parameters(e.g., time, repetition count, or alarm sound). The plurality of rulesmay include, e.g., one or more essential element parameters. Thematching rule may be stored in an NLU database (DB) 221.

According to an embodiment of the disclosure, the NLU module 220 mayidentify the meaning of a word extracted from the user input usinglinguistic features (e.g., syntactic elements) such as morphemes orphrases, match the identified meaning of the word to the domain andintent, and determine the user's intent. For example, the NLU module 220may calculate how many words extracted from the user input are includedin each domain and intent to thereby determine the user's intent.According to an embodiment of the disclosure, the NLU module 220 maydetermine the parameters of the user input using the word which is abasis for identifying the intent. According to an embodiment of thedisclosure, the NLU module 220 may determine the user's intent using theNLU DB 221 storing the linguistic features for identifying the intent ofthe user input. According to an embodiment of the disclosure, the NLUmodule 220 may determine the user's intent using a personal languagemodel (PLM). For example, the NLU module 220 may determine the user'sintent using personal information (e.g., contacts list or music list).The PLM may be stored in, e.g., the NLU DB 221. According to anembodiment of the disclosure, the ASR module 210, but not the NLU module220 alone, may recognize the user's speech by referring to the PLMstored in the NLU DB 221.

According to an embodiment of the disclosure, the NLU module 220 maygenerate a path rule based on the intent of the user input andparameters. For example, the NLU module 220 may select an app to beexecuted based on the intent of the user input and determine operationsto be performed on the selected app. The NLU module 220 may determineparameters corresponding to the determined operations to generate a pathrule. According to an embodiment of the disclosure, the path rulegenerated by the NLU module 220 may include information about the app tobe executed, operations (e.g., at least one or more states) to beexecuted on the app, and the parameters necessary to execute theoperations.

According to an embodiment of the disclosure, the NLU module 220 maygenerate one or more path rules based on the parameters and intent ofthe user input. For example, the NLU module 220 may receive a path ruleset corresponding to the user terminal 100 from the path planner module230, map the parameters and intent of the user input to the receivedpath rule set, and determine the path rule.

According to an embodiment of the disclosure, the NLU module 220 maydetermine the app to be executed, operations to be executed on the app,and parameters necessary to execute the operations based on theparameters and intent of the user input, thereby generating one or morepath rules. For example, the NLU module 220 may generate a path rule byarranging the app to be executed and the operations to be executed onthe app in the form of ontology or a graph model according to the userinput using the information of the user terminal 100. The generated pathrule may be stored through, e.g., the path planner module 230 in a pathrule database (PR DB) 231. The generated path rule may be added to thepath rule set of the PR DB 231.

According to an embodiment of the disclosure, the NLU module 220 mayselect at least one of a plurality of path rules generated. For example,the NLU module 220 may select the optimal one of the plurality of pathrules. As another example, the NLU module 220 may select a plurality ofpath rules when only some operations are specified based on theutterance. The NLU module 220 may determine one of the plurality of pathrules by the user's additional input.

According to an embodiment of the disclosure, the NLU module 220 maysend the path rule to the user terminal 100 at a request for the userinput. For example, the NLU module 220 may send one path rulecorresponding to the user input to the user terminal 100. As anotherexample, the NLU module 220 may send a plurality of path rulescorresponding to the user input to the user terminal 100. For example,when only some operations are specified based on the utterance, theplurality of path rules may be generated by the NLU module 220.

According to an embodiment of the disclosure, the path planner module230 may select at least one of the plurality of path rules.

According to an embodiment of the disclosure, the path planner module230 may deliver a path rule set including the plurality of path rules tothe NLU module 220. The plurality of path rules in the path rule set maybe stored in the form of a table in the PR DB 231 connected with thepath planner module 230. For example, the path planner module 230 maydeliver a path rule set corresponding to information (e.g., operatingsystem (OS) information or app information) of the user terminal 100which is received from the intelligent agent 145 to the NLU module 220.The table stored in the PR DB 231 may be stored, e.g., per domain or perdomain version.

According to an embodiment of the disclosure, the path planner module230 may select one or more path rules from the path rule set and deliverthe same to the NLU module 220. For example, the path planner module 230may match the user's intent and parameters to the path rule setcorresponding to the user terminal 100 to select one or more path rulesand deliver them to the NLU module 220.

According to an embodiment of the disclosure, the path planner module230 may generate one or more path rules using the user's intent andparameters. For example, the path planner module 230 may determine anapp to be executed and operations to be executed on the app based on theuser's intent and parameters to generate one or more path rules.According to an embodiment of the disclosure, the path planner module230 may store the generated path rule in the PR DB 231.

According to an embodiment of the disclosure, the path planner module230 may store the path rule generated by the NLU module 220 in the PR DB231. The generated path rule may be added to the path rule set stored inthe PR DB 231.

According to an embodiment of the disclosure, the table stored in the PRDB 231 may include a plurality of path rules or a plurality of path rulesets. The plurality of path rule or the plurality of path rule sets mayreflect the kind, version, type, or characteristic of the deviceperforming each path rule.

According to an embodiment of the disclosure, the DM module 240 maydetermine whether the user's intent identified by the path plannermodule 230 is clear. For example, the DM module 240 may determinewhether the user's intent is clear based on whether parameterinformation is sufficient. The DM module 240 may determine whether theparameters identified by the NLU module 220 are sufficient to perform atask. According to an embodiment of the disclosure, when the user'sintent is unclear, the DM module 240 may perform feedback to send arequest for necessary information to the user. For example, the DMmodule 240 may perform feedback to send a request for parameterinformation to identify the user's intent.

According to an embodiment of the disclosure, the DM module 240 mayinclude a content provider module. When the operation can be performedbased on the intent and parameters identified by the NLU module 220, thecontent provider module may generate the results of performing the taskcorresponding to the user input. According to an embodiment of thedisclosure, the DM module 240 may send the results generated by thecontent provider module to the user terminal 100 in response to the userinput.

According to an embodiment of the disclosure, the NLG module 250 mayconvert designated information into text. The text information may be inthe form of a natural language utterance. The designated information maybe, e.g., information about an additional input, information indicatingthat the operation corresponding to the user input is complete, orinformation indicating the user's additional input (e.g., feedbackinformation for the user input). The text information may be sent to theuser terminal 100 and displayed on the display 120, or the textinformation may be sent to the TTS module 260 and converted into aspeech.

According to an embodiment of the disclosure, the TTS module 260 mayconvert text information into speech information. The TTS module 260 mayreceive the text information from the NLG module 250, convert the textinformation into speech information, and send the speech information tothe user terminal 100. The user terminal 100 may output the speechinformation through the speaker 130.

According to an embodiment of the disclosure, the NLU module 220, thepath planner module 230, and the DM module 240 may be implemented in asingle module. For example, the NLU module 220, the path planner module230, and the DM module 240 may be implemented in a single module todetermine the user's intent and parameters and to generate a response(e.g., a path rule) corresponding to the user's intent and parametersdetermined. Accordingly, the generated response may be transmitted tothe user terminal 100.

FIG. 5 is a view illustrating an example of identifying current statesby a context module of an intelligent service module according to anembodiment of the disclosure.

Referring to FIG. 5, upon receiving (1) a context request from theintelligent agent 145, the processor 150 may send a request (2) forcontext information indicating the current states to the apps 141 and143 via the context module 149 a. According to an embodiment of thedisclosure, the processor 150 may, through the context module 149 a,receive (3) the context information from the apps 141 and 143 and send(4) to the intelligent agent 145.

According to an embodiment, the processor 150 may receive a plurality ofcontext information from the apps 141 and 143 via the context module 149a. The context information may be, e.g., information about the apps 141and 143 executed latest. As another example, the context information maybe information about the current states of the apps 141 and 143 (e.g.,in the case a photo is viewed in a gallery, information about thephoto).

According to an embodiment of the disclosure, the processor 150 may,through the context module 149 a, receive the context informationindicating the current state of the user terminal 100 not only from theapps 141 and 143 but also from a device platform. The contextinformation may include normal context information, user contextinformation, or device context information.

The normal context information may include normal information about theuser terminal 100. The normal context information may be identified byreceiving data through, e.g., a sensor hub of the device platform, andby an internal algorithm. For example, the normal context informationmay include information about the current time/space. The informationabout the current time/space may include information about, e.g., thecurrent time or the current position of the user terminal 100. Thecurrent time may be identified by the clock of the user terminal 100,and the information about the current position may be identified by theglobal positioning system (GPS). As another example, the normal contextinformation may include information about physical motion. Theinformation about physical motion may include information about, e.g.,walking, running, or driving. The physical motion information may beidentified by a motion sensor. The driving information may be identifiedby the motion sensor, and a Bluetooth connection in the vehicle may bedetected so that getting aboard and parking may be identified. Asanother example, the normal context information may include useractivity information. The user activity information may includeinformation about, e.g., commuting, shopping, or traveling. The useractivity information may be identified using information about the placewhich has been registered in the database by the user or app.

The user context information may include information about the user. Forexample, the user context information may include information about theuser's emotional state. The emotional state information may includeinformation about, e.g., the user's happiness, sadness, or anger. Asanother example, the user context information may include informationabout the user's current state. The current state information mayinclude information about, e.g., interest or intent (e.g., shopping).

The device context information may include information about the stateof the user terminal 100. For example, the device context informationmay include information about the path rule executed by the executionmanager module 147. As another example, the device information mayinclude battery information. The battery information may be identifiedthrough, e.g., the charged or discharged state of the battery. Asanother example, the device information may include information aboutthe network or a connected device. The information about the connecteddevice may be identified through the communication interface connectedwith the device.

FIG. 6 is a block diagram illustrating a proposal module of anintelligent service module according to an embodiment of the disclosure.

Referring to FIG. 6, the proposal module 149 c may include a hintproviding module 149 c_1, a context hint generating module 149 c_2, acondition checking module 149 c_3, condition model module 149 c_4, areuse hint generating module 149 c_5, or an introduction hint generatingmodule 149 c_6.

According to an embodiment of the disclosure, the processor 150 mayexecute the hint providing module 149 c_1 which may provide hints to theuser. For example, the processor 150 may, through the hint providingmodule 149 c_1, receive generated hints from the context hint generatingmodule 149 c_2, the reuse hint generating module 149 c_5, orintroduction hint generating module 149 c_6 and provide the hints to theuser.

According to an embodiment of the disclosure, the processor 150 mayexecute the condition checking module 149 c_3 or the condition modelmodule 149 c_4 to generate hits recommendable as per the current state.The processor 150 may execute the condition checking module 149 c_3 toreceive information corresponding to the current state and execute thecondition model module 149 c_4 to set a condition model using thereceived information. For example, the processor 150 may execute thecondition model module 149 c_4 to identify, e.g., the time, location,context, or app being used, when the hints are provided to the user, andprovide the user with the hints highly likely to be used in descendingorder of priority.

According to an embodiment of the disclosure, the processor 150 mayexecute the reuse hint generating module 149 c_5 to generate hints asper the use frequency. For example, the processor 150 may execute thereuse hint generating module 149 c_5 to generate hints based on theuser's use pattern.

According to an embodiment of the disclosure, the introduction hintgenerating module 149 c_6 may generate hints to introduce the user tonew functions or functions that other users frequently use. For example,a hint to introduce new functions may include an introduction to theintelligent agent 145 (e.g., a method to operate).

According to an embodiment of the disclosure, the context hintgenerating module 149 c_2, condition checking module 149 c_3, conditionmodel module 149 c_4, reuse hint generating module 149 c_5, orintroduction hint generating module 149 c_6 of the proposal module 149 cmay be included in the personal information server 300. For example, theprocessor 150 may, through the hint providing module 149 c_1 of theproposal module 149 c, receive hints from the context hint generatingmodule 149 c_2, reuse hint generating module 149 c_5, or introductionhint generating module 149 c_6 of the personal information server 300and provide the received hints to the user.

According to an embodiment of the disclosure, the user terminal 100 mayprovide hints according to a series of processes as follows. Forexample, upon receiving a provide hint request from the intelligentagent 145, the processor 150 may transfer a generate hint request to thecontext hint generating module 149 c_2 through the hint providing module149 c_1. Upon receiving the generate hint request, the processor 150may, through the condition checking module 149 c_3, receive informationcorresponding to the current state from the context module 149 a andpersona module 149 b. The processor 150 may transfer the receivedinformation to the condition model module 149 c_4 through the conditionchecking module 149 c_3 and may, the condition model module 149 c_4,assign priorities to the hints provided to the user in descending orderof availability using the information. The processor 150 may, throughthe context hint generating module 149 c_2, identify (6) the conditionand generate hints corresponding to the current state. The processor 150may transfer the generated hints to the hint providing module 149 c_1through the context hint generating module 149 c_2. The processor 150may, through the hint providing module 149 c_1, sort the hints as per adesignated rule and transfer the hints to the intelligent agent 145.

According to an embodiment of the disclosure, the processor 150 may,through the hint providing module 149 c_1, generate a plurality ofcontext hints and assign priorities to the plurality of context hints asper a designated rule. According to an embodiment of the disclosure, theprocessor 150 may, through the hint providing module 149 c_1, firstprovide the user with the higher-priority ones among the plurality ofcontext hints.

According to an embodiment of the disclosure, the user terminal 100 maypropose hints as per use frequency. For example, upon receiving aprovide hint request from the intelligent agent 145, the processor 150may transfer a generate hint request to the reuse hint generating module149 c_5 through the hint providing module 149 c_1. Upon receiving thegenerate hint request, the processor 150 may, through the reuse hintgenerating module 149 c_5, receive user information from the personamodule 149 b. For example, the processor 150 may, through the reuse hintgenerating module 149 c_5, receive the path rule included in the user'spreference information, parameters included in the path rule, frequencyof execution of app, and information about the time/space where the apphas been used, from the persona module 149 b. The processor 150 maygenerate hints corresponding to the received user information throughthe reuse hint generating module 149 c_5. The processor 150 may transferthe generated hints to the hint providing module 149 c_1 through thereuse hint generating module 149 c_5. The processor 150 may, through thehint providing module 149 c_1, sort the hints and transfer the hints tothe intelligent agent 145.

According to an embodiment of the disclosure, the user terminal 100 maypropose hints for new functions. For example, upon receiving a providehint request from the intelligent agent 145, the processor 150 maytransfer a generate hint request to the introduction hint generatingmodule 149 c_6 through the hint providing module 149 c_1. The processor150 may, through the introduction hint generating module 149 c_6,transfer a provide introduction hint request from a proposal server 400and receive information about functions to be introduced from theproposal server 400. For example, the proposal server 400 may store theinformation about the functions to be introduced. A hint list for thefunctions to be introduced may be updated by the service operator. Theprocessor 150 may transfer the generated hints to the hint providingmodule 149 c_1 through the introduction hint generating module 149 c_6.The processor 150 may, through the hint providing module 149 c_1, sortthe hints and transmit (6) the hints to the intelligent agent 145.

Accordingly, the processor 150 may provide hints generated by thecontext hint generating module 149 c_2, reuse hint generating module 149c_5, or introduction hint generating module 149 c_6 to the user throughthe proposal module 149 c. For example, the processor 150 may, throughthe proposal module 149 c, display the generated hints on the app thatoperates the intelligent agent 145 and receive inputs to select thehints from the user through the app.

FIG. 7 is a view illustrating a method for generating a path rule by apath planner module according to an embodiment of the disclosure.

Referring to FIG. 7, according to an embodiment of the disclosure, theNLU module 220 may sort functions of an app into any one operation(e.g., state A to state F) and store in the PR DB 231. For example, theNLU module 220 may store a path rule set including a plurality of pathrules A-B1-C1, A-B1-C2, A-B1-C3-D-F, and A-B1-C3-D-E-F divided into anyone operation in the PR DB 231.

According to an embodiment of the disclosure, the PR DB 231 of the pathplanner module 230 may store the path rule set to perform the functionsof the app. The path rule set may include a plurality of path rulesincluding the plurality of operations (e.g., a sequence of states). Inthe plurality of path rules, the operations executed as per theparameters each inputted to a respective one of the plurality ofoperations may sequentially be arranged. According to an embodiment ofthe disclosure, the plurality of path rules may be configured in theform of ontology or a graph model and stored in the PR DB 231.

According to an embodiment of the disclosure, the NLU module 220 mayselect the optimal one A-B1-C3-D-F of the plurality of path rulesA-B1-C1, A-B1-C2, A-B1-C3-D-F, and A-B1-C3-D-E-F corresponding to theparameters and the intent of the user input.

According to an embodiment of the disclosure, the NLU module 220 maydeliver the plurality of path rules to the user terminal 100 unlessthere is a path rule perfectly matching the user input. For example, theNLU module 220 may select the path rule (e.g., A-B1) partiallycorresponding to the user input. The NLU module 220 may select one ormore path rules (e.g., A-B1-C1, A-B1-C2, A-B1-C3-D-F, A-B1-C3-D-E-F)including the path rule (e.g., A-B1) partially corresponding to the userinput and deliver the same to the user terminal 100.

According to an embodiment of the disclosure, the NLU module 220 mayselect one of the plurality of path rules based on an additional inputof the user terminal 100 and deliver the selected path rule to the userterminal 100. For example, the NLU module 220 may select one (e.g.,A-B1-C3-D-F) among the plurality of path rules (e.g., A-B1-C1, A-B1-C2,A-B1-C3-D-F, A-B1-C3-D-E-F) as per an additional user input (e.g., aninput to select C3) of the user terminal 100 and send the selected pathrule to the user terminal 100.

According to an embodiment of the disclosure, the NLU module 220 maydetermine the user's intent and parameters corresponding to theadditional user input (e.g., an input to select C3) to the user terminal100 through the NLU module 220 and send the user's intent or parametersdetermined to the user terminal 100. The user terminal 100 may selectone (e.g., A-B1-C3-D-F) among the plurality of path rules (e.g.,A-B1-C1, A-B1-C2, A-B1-C3-D-F, A-B1-C3-D-E-F) based on the parameters orintent sent.

Accordingly, the user terminal 100 may complete the operations of theapps 141 and 143 by the selected path rule.

According to an embodiment of the disclosure, when a user input havinginsufficient information is received by the intelligent server 200, theNLU module 220 may generate a path rule partially corresponding to thereceived user input. For example, the NLU module 220 may send thepartially corresponding path rule to the intelligent agent 145. Theprocessor 150 may execute the intelligent agent 145 to receive the pathrule and transfer the partially corresponding path rule to the executionmanager module 147. The processor 150 may execute a first app 141 as perthe path rule through the execution manager module 147. The processor150 may, through the execution manager module 147, send informationabout the insufficient parameters to the intelligent agent 145 whileexecuting the first app 141. The processor 150 may, through theintelligent agent 145, send a request for additional input to the userusing the information about the insufficient parameters. Upon receivingan additional input from the user, the processor 150 may, through theintelligent agent 145, send the same to the intelligent server 200 forprocessing. The NLU module 220 may generate an added path rule based onthe parameter information and intent of the additional user input andsend the path rule to the intelligent agent 145. The processor 150 may,through the intelligent agent 145, send the path rule to the executionmanager module 147 to execute a second app 143.

According to an embodiment of the disclosure, when a user input havingsome missing information is received by the intelligent server 200, theNLU module 220 may send a request for user information to the personalinformation server 300. The personal information server 300 may send, tothe NLU module 220, information about the user who has entered the userinput stored in the persona database. The NLU module 220 may select apath rule corresponding to the user input having some missing operationsusing the user information. Accordingly, although a user input havingsome missing information is received by the intelligent server 200, theNLU module 220 may send a request for the missing information andreceive an additional input, or the NLU module 220 may use the userinformation, determining a path rule corresponding to the user input.

Table 1 below may represent an example path rule related to tasksrequested by the user according to an embodiment.

TABLE 1 Path rule ID State parameter Gallery_101 pictureView(25) NULLsearchView(26) NULL searchViewResult(27) Location, timeSearchEmptySelectedView(28) NULL SearchSelectedView(29) ContentType,selectall CrossShare(30) Anaphora

Referring to Table 1, a path rule generated or selected by anintelligent server (e.g., the intelligent server 200 of FIG. 1)according to an utterance (e.g., “Share photos”) may include at leastone state 25, 26, 27, 28, 29, or 30. For example, the at least one state(e.g., any one operation state of the terminal) may correspond to atleast one of executing a photo application (PicturesView) 25, executinga photo search function (SearchView) 26, outputting a search resultdisplay screen (SearchViewResult) 27, outputting a search result displayscreen with no photo selected (SearchEmptySelectedView) 28, outputting asearch result display screen with at least one photo selected(SearchSelectedView) 29, or outputting a shared application selectionscreen (CrossShare) 30.

According to an embodiment, the path rule parameter information maycorrespond to at least one state. For example, it may be included in thestate 29 of outputting a search result display screen with at least onephoto selected.

As a result of performing the path rule including the sequence of states25, 26, 27, 28, 29, or 30, the task (e.g., “Share photos!”) requested bythe user may be performed.

FIG. 8 is a view illustrating an example of managing user information bya persona module of an intelligent service module according to anembodiment of the disclosure.

Referring to FIG. 8, the processor 150 may receive information about theuser terminal 100 from the apps 141 and 143, execution manager module147, or the context module 149 a through the persona module 149 b. Theprocessor 150 may, through the apps 141 and 143 and the executionmanager module 147, store resultant information of execution of the appoperations 141 b and 143 b in the operation log database. The processor150 may, through the context module 149 a, store information about thecurrent state of the user terminal 100 in the context database. Theprocessor 150 may, through the persona module 149 b, receive the storedinformation from the operation log database or the context database. Thedata stored in the operation log database and the context database maybe analyzed by, e.g., an analysis engine, and transferred to the personamodule 149 b.

According to an embodiment of the disclosure, the processor 150 may,through the persona module 149 b, send the information received from theapps 141 and 143, the execution manager module 147, or the contextmodule 149 a to the proposal module 149 c. For example, the processor150 may, through the persona module 149 b, transfer the data stored inthe operation log database or context database to the proposal module149 c.

According to an embodiment of the disclosure, the processor 150 may,through the persona module 149 b, send the information received from theapps 141 and 143, the execution manager module 147, or the contextmodule 149 a to the personal information server 300. For example, theprocessor 150 may, through the persona module 149 b, periodically sendthe data accrued in the operation log database or context database tothe personal information server 300.

According to an embodiment, the processor 150 may, through the personamodule 149 b, transfer the data stored in the operation log database orcontext database to the proposal module 149 c. User informationgenerated by the persona module 149 b may be stored in a personadatabase. The persona module 149 b may periodically send userinformation stored in the persona database to the personal informationserver 300. According to an embodiment of the disclosure, theinformation sent by the persona module 149 b to the personal informationserver 300 may be stored in the persona database. The personalinformation server 300 may infer user information necessary to generatea path rule of the intelligent server 200 using the information storedin the persona database.

According to an embodiment of the disclosure, the user informationinferred using the information sent from the persona module 149 b mayinclude profile information or preference information. The profileinformation or the preference information may be inferred through theuser's account and accrued information.

The profile information may include the user's personal information. Theprofile information may include, e.g., the user's population statisticsinformation. The population statistics information may include, e.g.,the user's gender or age. As another example, the profile informationmay include life event information. The life event information may beinferred by comparing log information with a life event model, andreinforced by analyzing the behavior pattern. As another example, theprofile information may include interest information. The interestinformation may include, e.g., shopping goods of interest or field ofinterest (e.g., sports or politics). For example, the profileinformation may include activity region information. The activity regioninformation may include information about, e.g., a home or workplace.The activity region information may include not only information aboutplace or location but also information about regions recorded withpriorities based on the accrued time of stay and visit count. Forexample, the profile information may include activity time information.The activity time information may include information about, e.g.,wake-up time, commute time, or sleep time. The commute time informationmay be inferred using the activity region information (e.g., informationabout a home or workplace). The sleep time information may be inferredthrough unused time of the user terminal 100.

The preference information may include information about the user'spreference. For example, the preference information may includeinformation about an app preference. The app preference may be inferredthrough, e.g., record of use of an app (e.g., use record per time orplace). The app preference may be used to determine an app to beexecuted as per the user's current state (e.g., time or place). Forexample, the preference information may include information aboutcontacts preference. The contacts preference may be inferred byanalyzing, e.g., information about how frequent one has contact withanother, e.g., per time or place. The contacts preference may be used todetermine contacts as per the user's current state (e.g., contact tooverlapping names). As another example, the preference information mayinclude setting information. The setting information may be inferred by,e.g., analyzing how frequent particular settings are made, e.g., pertime or place. The setting information may be used to make particularsettings as per the user's current state (e.g., time, place, orcontext). For example, the preference information may include placepreference. The place preference may be inferred through, e.g., a recordof visits to a particular place (e.g., a record of visit per time). Theplace preference may be used to determine the place that the user isvisiting as per the user's current state (e.g., time). For example, thepreference information may include command preference. The commandpreference may be inferred through, e.g., frequency of use of commands(e.g., use frequency per time or place). The command preference may beused to determine a command pattern to be used as per the user's currentstate (e.g., time or place). In particular, the command preference mayinclude information about the menu chosen mostly by the user in thecurrent state of the app executed through analysis of log information.

FIG. 9 is a flowchart illustrating an example operation of executing afunction of an electronic device in an intelligence system according toan embodiment of the disclosure.

Referring to FIG. 9, according to an embodiment, an electronic device(e.g., the user terminal 100) may receive an utterance while performinga first operation in operation 910. For example, the first operation mayinclude a call event, an alarm event, a timer event, music play, videoplay, a hands-free event, or a content downloading event.

According to an embodiment, the electronic device may identify that theutterance includes a word selected to use a speech-based intelligentservice in operation 920. For example, the utterance may be a wake-uputterance used to invoke a speech-based intelligent service and may onlyinclude at least one selected wake-up word.

According to an embodiment, the electronic device may identify aresponse related to the first operation in operation 930. For example,information contained in the response may include information about thefirst operation, information about the progress of the first operation,or information about an operation subsequent to the first operation.

According to an embodiment, the electronic device may output theresponse through the speaker in operation 940. For example, the responsemay include the information about the first operation or the informationabout the progress of the first operation, which is contained in theresponse, or information inquiring whether to perform the subsequentoperation.

According to an embodiment, a method of executing a function of anelectronic device in an intelligence system comprises performing atleast one operation, receiving an utterance through a microphone of theelectronic device while performing the at least one operation,generating or selecting a response related to the first operation, andproviding the response through a speaker, wherein the utterance mayinclude only at least one selected wake-up word used to invoke aspeech-based intelligent service.

According to an embodiment, the method may further comprise transmittingdata related to the utterance to a server and receiving, from theserver, a response generated in relation with the at least oneoperation.

According to an embodiment, the method may further comprise identifyingprogress information varied or added within a designated time aftergenerating the response and further outputting a first responseincluding the identified progress information through the speaker.

According to an embodiment, the method may further comprise receiving auser input for the response, identifying a second operation related tothe first operation when the received user input corresponds to adesignated input, and further outputting a second response to identifywhether to perform the second operation.

According to an embodiment, the method may further comprise receiving auser input for the response and when the received user input does notcorrespond to a designated input, performing at least one functioncorresponding to the first operation.

According to an embodiment, the method may further comprise transmittingdata related to the first operation to the server when no additionalutterance is received within a designated time after the utterance isreceived.

According to an embodiment, the method may further comprise identifyingwhether the electronic device is operated in a designated mode and,after receiving the utterance, transmitting data related to the firstoperation to the server.

FIG. 10 is a flowchart illustrating an example operation of executing afunction of an electronic device in an intelligence system according toan embodiment of the disclosure.

Referring to FIG. 10, according to an embodiment, an electronic device(e.g., the user terminal 100) may identify entry of a wake-up command(e.g., a wake-up utterance or hardware key entry) from the user inoperation 1010. For example, the wake-up utterance may be constituted ofdesignated words (e.g., “Bixby!”).

According to an embodiment, the electronic device may identify that awake-up command is entered by identifying selection of a designatedbutton (e.g., a hardware key 112) of the electronic device or a buttonof an external electronic device (e.g., a wired or wireless earphone)wired or wirelessly connected with the electronic device.

According to an embodiment, the electronic device may identify taskinformation about an application running on the electronic device inoperation 1020. For example, the task information may includeinformation related to an operation that may be provided from therunning application.

According to an embodiment, the electronic device may output a firstfeedback for the utterance based on the task information in operation1030. For example, the first feedback may include information about anoperation being executed or an event generated from the application orinformation inquiring whether to perform a first task that may beperformed as an operation subsequent to the operation being executed orthe event.

According to an embodiment, when the electronic device receives a call,the electronic device may identify an operation of taking the call asthe first task.

According to an embodiment, the electronic device may determine whethera positive input for the first feedback is received in operation 1040.For example, the input for the first feedback may include a positive ornegative response as to whether to perform the first task. According toan embodiment, unless the input is received within a designated time,the electronic device may identify it as a negative response.

According to an embodiment, when a positive response to the firstfeedback is received as a result of performing operation 1040 describedabove, the electronic device may perform the first task in operation1041.

According to an embodiment, when a negative response to the firstfeedback is received as a result of performing operation 1040 describedabove, the electronic device may identify a second task related to thefirst task in operation 1050. For example, the second task may includeinformation related to an operation performed as an operation subsequentto the first task on the application.

According to an embodiment, the second task may be an operation oftransmitting a message to the other party whose call has been rejected,as an operation performable as an operation subsequent to the firsttask. For example, according to reception of a negative response for thefirst task, the electronic device may determine that the user may notreceive the call and identify the second task as an operation oftransmitting, to the other party, a message containing information(e.g., “Cannot take a call.”) indicating that the user may not receivethe call.

According to an embodiment, the electronic device may output a secondfeedback related to the second task in operation 1051. For example, thesecond feedback may include information (e.g., “Message will betransmitted to the other party.”) about an operation performed on theelectronic device as the second task is performed.

According to an embodiment, the electronic device may perform the secondtask in operation 1052. For example, unless a response is receivedwithin a designated time after the second feedback is output, theelectronic device may identify it as receiving a positive response forthe second feedback and perform the second task.

FIG. 11 is a flowchart illustrating an example operation between aserver and an electronic device in an intelligence system according toan embodiment of the disclosure.

Referring to FIG. 11, an intelligence system may include an intelligentserver 200 and a user terminal 100.

According to an embodiment, an intelligent agent 145 of a user terminal100 may identify entry of a wake-up command (e.g., a wake-up utterance)in operation 1110. For example, the wake-up utterance may be constitutedof designated words (e.g., “Bixby!”).

According to an embodiment, the electronic device may identify that thewake-up utterance is entered by identifying selection of a designatedbutton (e.g., the hardware key 112 of FIG. 3) of the electronic deviceor a button of an external electronic device (e.g., a cover or wired orwireless earphone) wired or wirelessly connected with the electronicdevice.

According to an embodiment, the intelligent agent 145 may determinewhether an additional utterance is entered within a predetermined timeafter the utterance has been entered in operation 1120. When anadditional utterance is determined to be entered, an execution managermodule 147 of a user terminal 100 may deliver an operation executioncommand related to the utterance to an application to execute anoperation of the application related to the utterance in operation 1121.

According to an embodiment, when no additional utterance is determinedto be entered as a result of operation 1120, the intelligent agent 145may send a request for application context information to a firstapplication 141 in operation 1130.

According to an embodiment, the first application 141 may identifycontext information about an event generated from the first application141 in operation 1140. For example, the context information may includeinformation related to a call event from ‘Suzie,’ who is a ‘colleague’,as the call event is generated.

According to an embodiment, the first application 141 may transmit thecontext information of the application to the intelligent agent 145 inoperation 1150.

According to an embodiment, the intelligent agent 145 may transmit thecontext information to the intelligent server 200 in operation 1151.

According to an embodiment, the intelligent server 200 may transmit aninformation feedback for the utterance to the intelligent agent 145 inoperation 1160.

According to an embodiment, the intelligent agent 145 may performcontrol to allow the transmitted feedback to be output through aninput/output device of the user terminal 100. For example, the feedbackmay contain the content “You have a call from your colleague, Suzie,” asinformation indicating that a call is received from Suzie who is acolleague.

FIG. 12 is a block diagram illustrating example components of anintelligence system and example operations among the componentsaccording to an embodiment of the disclosure.

Referring to FIG. 12, an intelligence system may include an intelligentserver 200 and a user terminal 100. The intelligent server 200 mayinclude an ASR module 210, an NLU module 220, and a TTS module 260, andthe user terminal 100 may include an intelligent agent 145, an executionmanager module 147, and a phone application (e.g., a dialer 1253).

According to an embodiment, the dialer 1253 may identify that a callevent occurs in operation 1210. For example, when the call event occurs,the dialer 1253 may output a ringtone and caller information in varioustypes of data, such as a speech, image, or text data, thereby indicatingthat the call is received.

According to an embodiment, the intelligent agent 145 may identify entryof a wake-up command (or wake-up utterance) from the user while the callevent occurs in operation 1220. For example, the wake-up utterance maybe constituted of designated words and may be a speech command toactivate an intelligent service.

According to an embodiment, the intelligent agent 145 may determinewhether an additional utterance is entered within a predetermined timeafter the wake-up utterance has been entered in operation 1230. Forexample, when an additional utterance is entered within the designatedtime, an operation of an application of the user terminal 100corresponding to the additional utterance may be performed.

According to an embodiment, upon determining that no additionalutterance is entered within the designated time, the intelligent agent145 may control the execution manager module 147 to send a request forcontext information about a phone application to the dialer 1253 inoperation 1240.

According to an embodiment, the dialer 1253 may identify the contextinformation of the phone application and transfer to the executionmanager module 147 in operation 1250.

According to an embodiment, the intelligent agent 145 may transfer thecontext information to the ASR module 210 of the intelligent server 200in operation 1251.

According to an embodiment, the ASR module 210 of the intelligent server200 may convert the utterance into text data and transfer the text datato the NLU module 220 in operation 1252.

According to an embodiment, the NLU module 220 may select feedback forthe utterance based on the context information in operation 1260. Forexample, depending on whether a subsequent task or progress informationis identified in relation to the call event, the NLU module 220 mayidentify information to be contained in the feedback for the utteranceamong various pieces of information.

According to an embodiment, the intelligent server 200 may transmit theselected feedback via the TTS module 260 to the user terminal 100 inoperation 1270.

According to an embodiment, the intelligent agent 145 may outputfeedback for the utterance in operation 1280.

FIG. 13 is a flowchart illustrating an example operation between aserver and an electronic device in an intelligence system according toan embodiment of the disclosure.

Referring to FIG. 13, an intelligence system may include an intelligentserver 200 and a user terminal 100. The user terminal 100 may include anintelligent agent 145 and a first application 141.

According to an embodiment, the user terminal 100 may identify that anevent occurs through the first application 141. For example, the firstapplication 141 may be an application for playing content and mayidentify an occurrence of an event of downloading a content file to beplayed.

According to an embodiment, the intelligent agent 145 of the userterminal 100 may identify entry of a wake-up command (e.g., a wake-uputterance) while the event occurs in operation 1310. For example, thewake-up utterance may be constituted of designated words (e.g.,“Bixby!”).

According to an embodiment, the intelligent agent 145 may identify thatthe wake-up utterance is entered by identifying selection of adesignated button (e.g., the hardware key 112) of the user terminal 100or a button of an external electronic device (e.g., a wired or wirelessearphone) wired or wirelessly connected with the user terminal 100.

According to an embodiment, the intelligent agent 145 may determinewhether an additional utterance is entered within a predetermined timeafter the utterance has been entered in operation 1320.

According to an embodiment, when an additional utterance is determinedto be entered with the predetermined time in operation 1320, theintelligent agent 145 may transfer a command for executing an operationrelated to the additional utterance to the first application 141 inoperation 1321. For example, although it is shown that the intelligentagent 145 transfers the command for executing the operation related tothe additional utterance to the first application 141, when theadditional utterance indicates an operation of an application differentfrom the first application 141, the intelligent agent 145 may performcontrol to allow the operation execution command to be transferred tothe application.

According to an embodiment, when no additional utterance is determinedto be entered within the predetermined time in operation 1320, theintelligent agent 145 may send a request for context information of thefirst application 141 to the execution manager module 147 in operation1330.

According to an embodiment, the execution manager module 147 maytransfer the context information request to the first application 141 inoperation 1331.

According to an embodiment, the first application 141 may identifycontext information related to the occurring event in operation 1340.

According to an embodiment, the first application 141 may transfer firstcontext information related to the occurring event to the intelligentagent 145 in operation 1350.

According to an embodiment, the intelligent agent 145 may transmit thefirst context information to the intelligent server 200 in operation1351.

According to an embodiment, the intelligent server 200 may identifyfeedback for utterance based on the transmitted application contextinformation in operation 1360. For example, the feedback may includecontent information and content download progress information asinformation about the event.

According to an embodiment, the intelligent server 200 may transmit theidentified feedback to the intelligent agent 145 in operation 1370.

According to an embodiment, the intelligent agent 145 may output thefeedback in various forms through the input/output device 340 (e.g., aspeaker or display) of the user terminal 100. For example, the feedbackmay include the content “(content) 10% downloaded” as event progressinformation or “music being downloaded” as event information.

According to an embodiment, the first application 141 may drive a timerin operation 1380. For example, the first application 141 may drive thetimer to determine whether second context information added or varied isidentified within a predetermined time.

According to an embodiment, upon identifying the second contextinformation before the timer expires, the first application 141 maytransfer the second context information to the execution manager module147 in operation 1381.

According to an embodiment, the execution manager module 147 maytransfer the second context information to the intelligent agent 145 inoperation 1382.

According to an embodiment, the intelligent agent 145 may transfer thesecond context information to the intelligent server 200 in operation1383.

According to an embodiment, the intelligent server 200 may identify anadditional feedback for the utterance based on the identified secondcontext information and transmit it to the intelligent agent 145 inoperation 1390. For example, the intelligent agent 145 may output theadditional feedback through the input/output device 340. The additionalfeedback may include the content “(content) 30% downloaded” asinformation indicating the varied content download has progressed.

According to an embodiment, when an utterance is entered while an eventfor an application occurs, the user terminal 100 may output, asfeedback, information indicating the progress of the event in relationto the utterance. Additionally, as the progress varies, the informationindicating the varied progress may be output as an additional feedback.For example, the user may receive, through the intelligence system, theprogress information about the event occurring for the application andthe varied progress information.

FIG. 14 is a flowchart illustrating an example operation between aserver and an electronic device in an intelligence system according toan embodiment of the disclosure.

Referring to FIG. 14, an intelligence system may include an intelligentserver 200 and a user terminal 100. The user terminal 100 may include anintelligent agent 145 and a dialer application 1253 (or a ‘dialer’).

According to an embodiment, the user terminal 100 may identify that acall event occurs through the dialer application in operation 1410.

According to an embodiment, the intelligent agent 145 of the userterminal 100 may identify entry of a wake-up command (e.g., a wake-uputterance) while the call event occurs in operation 1420. For example,the wake-up utterance may be constituted of designated words (e.g.,“Bixby!”).

According to an embodiment, the intelligent agent 145 may identify thatthe wake-up utterance is entered by identifying selection of adesignated button (e.g., the hardware key 112) of the user terminal 100or a button of an external electronic device (e.g., a wired or wirelessearphone) wired or wirelessly connected with the user terminal 100.

According to an embodiment, the intelligent agent 145 may send a requestfor context information of the phone application to the dialer 1253 inoperation 1430.

According to an embodiment, the dialer 1253 may transmit phoneapplication context information related to the occurring call event tothe intelligent agent 145 in operation 1440.

According to an embodiment, the intelligent server 200 may identify afirst feedback for utterance based on the transmitted phone applicationcontext information in operation 1450 and transmit data of the firstfeedback for utterance to the intelligent agent 145 in operation 1460.For example, the first feedback may include information about the callevent and information inquiring whether to perform an operationsubsequent to the event.

According to an embodiment, the intelligent agent 145 may output thefeedback in various forms through the input/output device 340 (e.g., aspeaker or display) of the user terminal 100 in operation 1461. Forexample, the feedback may include the content “You have a call from yourcolleague, Suzie,” as the call event-related information and the content“You want to take the call?” as the information inquiring whether toperform the subsequent operation.

According to an embodiment, the intelligent agent 145 may identify theuser's response to the first feedback and transmit the user's responseto the intelligent server 200 in operation 1470.

According to an embodiment, the intelligent server 200 may send arequest for a task of the phone application executable depending on theuser's response to the intelligent agent 145 in operation 1480. Forexample, when a negative user response for the first feedback is enteredor no response is received within a designated time, the intelligentserver 200 may send a request for a task of the phone applicationexecutable on the dialer 1253 as the additional operation to the callevent.

According to an embodiment, the intelligent agent 145 may transfer thetask request to the dialer 1253 in operation 1481.

According to an embodiment, the dialer 1253 may identify a taskaccording to the user's response and transmit the identified task to theintelligent server 200 in operation 1482.

According to an embodiment, the intelligent server 200 may identify asecond feedback for the utterance based on the identified task andtransmit it to the intelligent agent 145 in operation 1490. For example,the second feedback may include information inquiring whether to executethe identified task.

According to an embodiment, as the user enters a negative response tothe utterance, e.g., not taking the call, for the occurring call event,the intelligent server 200 may identify, as the second feedback, theoperation of transmitting a message to notify the caller that the callmay not be taken, which corresponds to the subsequent operation of theutterance.

According to an embodiment, the intelligent agent 145 may output thesecond feedback received from the intelligent server 200 through theinput/output device 340 in operation 1491. For example, the secondfeedback may include the content “Do you want to send Suzie a message toindicate that you are driving?” to inquire whether to send a message tothe caller.

According to an embodiment, when an utterance is entered while an eventfor an application occurs, the user terminal 100 may output, asfeedback, information inquiring whether to perform the operation of theapplication where the event occurs in relation to the utterance.Selectively, the user terminal 100 may receive a user input for thefeedback and output information inquiring whether to additionallyperform an operation of the application. For example, the user mayidentify the occurring event on the application through the intelligencesystem and receive feedback inquiring whether to perform theevent-related operation.

FIG. 15 is a block diagram illustrating example components of anintelligence system and example operations among the componentsaccording to an embodiment of the disclosure.

Referring to FIG. 15, an intelligence system may include an intelligentserver 200 and a user terminal 100. The intelligent server 200 mayinclude an ASR module 210, an NLU module 220, and a TTS module 260, andthe user terminal 100 may include an intelligent agent 145, an executionmanager module 147, and an alarm application 1560 b (or, an ‘alarm’).

According to an embodiment, the alarm 1560 b may identify that an alarmevent occurs in operation 1510. For example, when the alarm eventoccurs, the alarm 1560 b may output information to indicate that thecurrent time has reached a particular date and time or that a designatedschedule is coming up in various types of data, e.g., speech, image, ortext data.

According to an embodiment, the intelligent agent 145 may identify entryof a wake-up command (or wake-up utterance) from the user while thealarm event occurs in operation 1520. For example, the wake-up utterancemay be constituted only of designated words.

According to an embodiment, the intelligent agent 145 may determinewhether an additional utterance is entered within a predetermined timeafter the utterance has been entered in operation 1521. For example,when an additional utterance is entered within the designated time, anoperation of an application of the user terminal 100 corresponding tothe additional utterance may be performed.

According to an embodiment, upon determining that no additionalutterance is entered within the designated time, the intelligent agent145 may control the execution manager module 147 to send a request forcontext information of an alarm application to the alarm 1560 b inoperation 1530.

According to an embodiment, the alarm 1560 b may identify the contextinformation of the alarm application and transfer to the executionmanager module 147 in operation 1531.

According to an embodiment, the execution manager module 147 maytransmit the context information to the intelligent server 200. Forexample, the NLU module 220 may select a first feedback for theutterance based on the context information in operation 1540 andtransmit the selected first feedback to the user terminal 100 throughthe TTS module 260 in operation 1544.

According to an embodiment, the intelligent agent 145 may output thefirst feedback for the utterance in operation 1550.

According to an embodiment, the intelligent agent 145 may identify auser input for the first feedback in operation 1551. For example, theintelligent agent 145 may determine whether the user input indicates apositive or negative response to the first feedback or is receivedwithin a designated time.

According to an embodiment, the intelligent agent 145 may transfer theuser input to the first feedback to the alarm 1560 b in operation 1560.

According to an embodiment, the alarm 1560 b may identify second contextinformation according to task information as per the user input for thefirst feedback and transfer it to the intelligent agent 145 in operation1561. For example, when the user input indicates a negative response oris not received within a designated time, the alarm 1560 b may identifysecond context information based on task flow (1563 a) information inthe alarm 1560 b.

According to an embodiment, in operation 1562, the intelligent agent 145may transmit the second context information to the ASR module 210 of theintelligent server 200 and reperform operations 1540 and 1544, selectingthe second feedback for the utterance based on the second contextinformation through the NLU module 220 and transmitting the selectedfeedback to the user terminal 100 through the TTS module 260.

According to an embodiment, the task flow 1563 a may include informationabout an operation flow when an event occurs in the alarm 1560 b, andinformation inquiring whether to perform a particular operation of theoperation flow may be output as feedback. For example, the alarm 1560 bmay include a task flow including the operations of going off, turningoff, or snoozing an alarm when an alarm event occurs.

According to an embodiment, the intelligent agent 145 may output thesecond feedback through an input/output device (e.g., the display 120 orspeaker 130 of FIG. 2) of the user terminal 100 in operation 1570.

FIG. 16 is a view illustrating an example screen configuration ofsetting an application to use an intelligent service according to anembodiment of the disclosure.

Referring to FIG. 16, a user terminal 100 may display a screen 1601 toset up whether to support feedback depending on the execution state inusing an intelligent service. The screen 1601 may be displayed when aparticular menu item of an application is selected or otherwise when adesignated input is received.

According to an embodiment, the screen 1601 may display a list ofapplications and a function setting button 1611 for each application.The list of applications may include a phone application (call) 1610, amusic play application (music), an alarm application (alarm), an instantmessaging application (IM or K-talk), a short message service/multimediamessaging service (MMS) application (message), or a video playapplication (video) or other various applications stored in the userterminal 100.

According to an embodiment, when an input on a button 1611 correspondingto the ‘call’ 1610 is received, the button 1611 may be moved, e.g.,right, and be set to ‘on.’ In this case, feedback for an enteredutterance may be output based on the execution state of the alarmapplication. For example, when a wake-up utterance is entered while acall event occurs in the phone application, feedback for the enteredutterance may be output including the caller information included in thecall event or information inquiring whether to perform a subsequentoperation (e.g., call reception).

According to an embodiment, when the ‘call’ 1610 item of the applicationlist is selected, the screen 1601 may switch into a screen 1602 toselect a detailed feedback type of ‘call.’ For example, the screen 1602may include a selection item (e.g., 1621 a) to select feedback to beoutput and the type of feedback.

According to an embodiment, types of feedback may include a firstfeedback (information feedback) 1621 (e.g., “You have a call.”)including information about the event, a second feedback (next orsubsequent task feedback) 1622 (e.g., “Do you want to take the call?”)including information about a task performable as an operationsubsequent to the event, or a third feedback (progress feedback) 1623(e.g., “Phone has rung three times.”) including information related tothe progress of the event For example, when the user selects theselection item 1621 a for the first feedback 1621, and an utterance isentered while a call event occurs in the phone application, feedbackcontaining information about the call event may be output as feedbackfor the utterance.

FIGS. 17A, 17B, and 17C are views illustrating various example scenariosof outputting feedback by an electronic device according to entry of auser's utterance in an intelligence system according to an embodiment ofthe disclosure.

Referring to FIG. 17A, an alarm event may be occurring from an alarmapplication (e.g., the alarm 1560 b) in a user terminal 100. The user1301 may enter a wake-up utterance (e.g., “Bixby!”) to the user terminal100 while the alarm event occurs.

According to an embodiment, when there is no additional utterance withina designated time after the utterance has been entered, the userterminal 100 may identify context information (app context info.) of thealarm application where the event occurs. The context information of thealarm application may include a current task, a subsequent task, orfeedback type (speech feedback type). For example, the user terminal 100may identify the current task as alarm ringing, the subsequent task asan alarm end task as an operation performable as an operation subsequentto the current task, and the feedback type as feedback includinginformation inquiring whether to perform the ‘next task’ as thesubsequent task is identified.

According to an embodiment, the user terminal 100 may output feedbackfor the utterance depending on the feedback type. For example, the userterminal 100 may output the feedback including information (e.g., “Doyou want to turn off the alarm?”) inquiring whether to end the alarm asthe task subsequent to the alarm ringing operation which is the currenttask.

According to an embodiment, the user may activate the intelligencesystem using a brief word and not only identify the operation subsequentto the currently occurring event but also determine whether to perform.

Referring to FIG. 17B, a call event may occur from a phone application(e.g., the dialer 1253) in the user terminal 100. The user 1701 mayenter a wake-up utterance (e.g., “Bixby!”) to the user terminal 100while the call event occurs.

According to an embodiment, when there is no additional utterance withina designated time after the utterance has been entered, the userterminal 100 may identify context information of the phone applicationwhere the event occurs. The current task of the phone application may beidentified as call receiving (call ringing), and the feedback type maybe identified as feedback including information about the current task.

According to an embodiment, the user terminal 100 may output informationabout the current task executed on the phone application as feedback forthe utterance depending on the feedback type. For example, the userterminal 100 may output, as the feedback for the utterance, feedbackincluding the content “You have a call from Suzie.”

According to an embodiment, the user terminal 100 may additionallyidentify a task subsequent to the current task or output feedbackincluding information inquiring whether to perform the subsequent taskand the information about the current task depending on the feedbacksettings for the phone application. For example, as the user terminal100 identifies the operation of taking the received call as thesubsequent task to the current task of call receiving, the feedback forthe utterance may include the content “You have a call from Suzie. Doyou want to take the call?”

According to an embodiment, the user may activate the intelligencesystem using a brief word and receive, as feedback, information aboutthe currently occurring event and information inquiring whether toperform the subsequent operation.

Referring to FIG. 17C, a music download event may occur from a musicplay application (e.g., the media player 382) in the user terminal 100.The user 1701 may enter a wake-up utterance (e.g., “Bixby!”) to the userterminal 100 while the music download event occurs.

According to an embodiment, when there is no additional utterance withina designated time after the utterance has been entered, the userterminal 100 may identify context information of the music playapplication where the event occurs. For example, by identifying musicdownloading as the current task of the music play application, the userterminal 100 may identify the feedback type as outputting progressindicating information.

According to an embodiment, the user terminal 100 may output, asfeedback for the utterance, information about the music downloading taskexecuted on the music play application. For example, the user terminal100 may output feedback containing the content “music downloading 10%complete” as the feedback for the utterance.

According to an embodiment, as the user terminal 100 identifiesinformation (e.g., musician or song title) about the content where anevent occurs, the user terminal 100 may output, as feedback, informationabout the content, along with information indicating the progress of thecurrent task. For example, the feedback for the utterance may includethe content “Song by musician (e.g., TEE, Bye′) titled (Yawn) isdownloading. 10% complete.”

According to an embodiment, the user may activate the intelligencesystem using a brief word and receive, as feedback, information aboutthe currently occurring event and the progress.

FIG. 18 is a view illustrating an example scenario of outputtingfeedback from a smart speaker according to entry of an utterance in anintelligence system according to an embodiment of the disclosure.

Referring to FIG. 18, an intelligence system may include a smart speaker(or AI speaker) 1803, an intelligent service server 1804, or anintelligent server 200.

According to an embodiment, a user 1801 may enter a wake-up utterance tothe smart speaker 1803. For example, the smart speaker 1803 may be adevice connected with the intelligent server 200 to output feedback foran utterance received in the form of a speech.

According to an embodiment, an intelligent agent (e.g., the intelligentagent 145) of the smart speaker 1803 may identify that the wake-uputterance has been entered as it identifies selection of a designatedbutton (e.g., the hardware key 112) of the smart speaker 1803.

According to an embodiment, the smart speaker 1803 may transfer thereceived utterance to the intelligent service server 1804 that supportsan AI service platform for providing services through the smart speaker1803. For example, the intelligent service server 1804 may identifywhether a subsequent utterance comes in within a designated time.

According to an embodiment, although the smart speaker 1803 has beendescribed to transfer the received utterance to the intelligent serviceserver 1804, the intelligent service server 1804 may be omitted or, whennot connected with the smart speaker 1803, the utterance may betransferred to the intelligent server 200.

According to an embodiment, when no utterance subsequent to the receivedutterance is identified, the intelligent server 200 or the intelligentservice server 1804 may identify feedback based on the service beingcurrently executed on the smart speaker 1803 and the feedback may betransferred to the smart speaker 1803.

According to an embodiment, when the user 1801 issues an utterance,saying “Order a kitchen towel,” along with a wake-up utterance (e.g.,“Bixby˜”), the intelligent server 200 or the intelligent service server1804 may interpret the utterance of the user 1801 and perform anoperation for ordering a ‘kitchen towel.’ For example, the intelligentserver 200 or the intelligent service server 1804 may transfer a resultof performing the operation to the smart speaker 1803 so that the resultmay be outputted through the smart speaker 1803 in the form of a speech.

According to an embodiment, when no feedback for the received utteranceis output, such as when communication between the intelligent server 200or the intelligent service server 1804 and the smart speaker 1803 isdelayed, the user 1801 may reenter a wake-up utterance. For example, theintelligent server 200 or the intelligent service server 1804 maytransfer feedback for the reentered wake-up utterance includinginformation about the received utterance to the smart speaker 1803. Forexample, the feedback for the reentered wake-up utterance may includeinformation about the progress (e.g., “A search is now in progress.”) orinformation about a subsequent task (e.g., “Do you want to stopsearching?”) for the received utterance.

According to an embodiment, the user 1801 may play music as a streamingservice through the smart speaker 1803. For example, when the user 1801enters a wake-up utterance during a streaming service, the intelligentserver 200 or the intelligent service server 1804 may identifyinformation related to the streaming service as feedback for the wake-uputterance.

According to an embodiment, when the smart speaker 1803 temporarilydrops out while playing music or connection with the server providingthe streaming service is unstable, the user 1801 may enter a wake-uputterance (e.g., “Bixby˜”) to the smart speaker 1803. For example, theintelligent server 200 or the intelligent service server 1804 mayidentify information related to the streaming service being executed onthe smart speaker 1803 as feedback for the wake-up utterance. Thestreaming service-related information may include at least part ofconnection state information of the streaming service (e.g., “Musicdownloading is currently delayed due to network latency.”), subsequenttask information (e.g., “Do you want to play the downloaded music?” or“Do you want to play music via another music streaming service?”), orthe state of the streaming service (e.g., “Slow response from the server(e.g., intelligent service server 1804)”).

According to an embodiment, while playing a song on the streamingservice, the user 1801 may enter an utterance (e.g., “Save (or purchase)the song now in play.”) to store the song in the memory of the smartspeaker 1803 or purchase the song, along with a wake-up utterance (e.g.,“Bixby˜”), to the smart speaker 1803. For example, while a certain songis being downloaded based on the utterance, the user may enter a wake-uputterance to identify the state of the download. For example, the smartspeaker 1803 may output download state information (e.g., “Currently 30%downloaded. The remaining time is 3 seconds.”) as feedback for thewake-up utterance.

According to an embodiment, when the user enters a wake-up utterancethrough various electronic devices connected with the intelligent server200, feedback containing operation state information or subsequent taskinformation may be output based on a function executed on acorresponding electronic device.

FIG. 19 is a block diagram illustrating an electronic device 1901 (or afirst electronic device) in a network environment 1900 according tovarious embodiments of the disclosure.

Referring to FIG. 19, the electronic device 1901 (e.g., the userterminal 100 of FIG. 1) in the network environment 1900 may communicatewith an electronic device 1902 (or a second electronic device) via afirst network 1998 (e.g., a short-range wireless communication network),or an electronic device 1904 or a server 1908 (e.g., the intelligentserver 200 of FIG. 1) via a second network 1999 (e.g., a long-rangewireless communication network). According to an embodiment, theelectronic device 1901 may communicate with the electronic device 1904via the server 1908. According to an embodiment, the electronic device1901 may include a processor 1920 (e.g., the processor 150 FIG. 2),memory 1930 (e.g., the memory 140 of FIG. 2), an input device 1950, asound output device 1955 (e.g., the speaker 130 of FIG. 2), a displaydevice 1960 (e.g., the display 120 of FIG. 2), an audio module 1970, asensor module 1976, an interface 1977, a haptic module 1979, a cameramodule 1980, a power management module 1988, a battery 1989, acommunication module 1990, a subscriber identification module (SIM)1996, or an antenna module 1997. In some embodiments, at least one(e.g., the display device 1960 or the camera module 1980) of thecomponents may be omitted from the electronic device 1901, or one ormore other components may be added in the electronic device 101. In someembodiments, some of the components may be implemented as singleintegrated circuitry. For example, the sensor module 1976 (e.g., afingerprint sensor, an iris sensor, or an illuminance sensor) may beimplemented as embedded in the display device 1960 (e.g., a display).

The processor 1920 may execute, for example, software (e.g., a program1940) to control at least one other component (e.g., a hardware orsoftware component) of the electronic device 1901 coupled with theprocessor 1920, and may perform various data processing or computation.According to one embodiment, as at least part of the data processing orcomputation, the processor 1920 may load a command or data received fromanother component (e.g., the sensor module 1976 or the communicationmodule 1990) in volatile memory 1932, process the command or the datastored in the volatile memory 1932, and store resulting data innon-volatile memory 1934. According to an embodiment, the processor 1920may include a main processor 1921 (e.g., a CPU or an AP), and anauxiliary processor 1923 (e.g., a graphics processing unit (GPU), animage signal processor (ISP), a sensor hub processor, or a communicationprocessor (CP)) that is operable independently from, or in conjunctionwith, the main processor 1921. Additionally or alternatively, theauxiliary processor 1923 may be adapted to consume less power than themain processor 1921, or to be specific to a specified function. Theauxiliary processor 1923 may be implemented as separate from, or as partof the main processor 1921.

The auxiliary processor 1923 may control at least some of functions orstates related to at least one component (e.g., the display device 1960,the sensor module 1976, or the communication module 1990) among thecomponents of the electronic device 1901, instead of the main processor1921 while the main processor 1921 is in an inactive (e.g., sleep)state, or together with the main processor 1921 while the main processor1921 is in an active state (e.g., executing an application). Accordingto an embodiment, the auxiliary processor 1923 (e.g., an ISP or a CP)may be implemented as part of another component (e.g., the camera module1980 or the communication module 1990) functionally related to theauxiliary processor 1923.

The memory 1940 may store various data used by at least one component(e.g., the processor 1920 or the sensor module 1976) of the electronicdevice 1901. The various data may include, for example, software (e.g.,the program 1940) and input data or output data for a command relatedthereto. The memory 1940 may include the volatile memory 1932 or thenon-volatile memory 1934.

The program 1940 may be stored in the memory 1930 as software, and mayinclude, for example, an OS 1942, middleware 1944, or an application1946.

The input device 1950 may receive a command or data to be used byanother component (e.g., the processor 1920) of the electronic device1901, from the outside (e.g., a user) of the electronic device 1901. Theinput device 1950 may include, for example, a microphone, a mouse, or akeyboard.

The sound output device 1955 may output sound signals to the outside ofthe electronic device 1901. The sound output device 1955 may include,for example, a speaker or a receiver. The speaker may be used forgeneral purposes, such as playing multimedia or playing records, and thereceiver may be used for an incoming call. According to an embodiment,the receiver may be implemented as separate from, or as part of thespeaker.

The display device 1960 may visually provide information to the outside(e.g., a user) of the electronic device 1901. The display device 1960may include, for example, a display, a hologram device, or a projectorand control circuitry to control a corresponding one of the display,hologram device, and projector. According to an embodiment, the displaydevice 1960 may include touch circuitry adapted to detect a touch, orsensor circuitry (e.g., a pressure sensor) adapted to measure theintensity of force incurred by the touch.

The audio module 1970 may convert a sound into an electrical signal andvice versa. According to an embodiment, the audio module 1970 may obtainthe sound via the input device 1950, or output the sound via the soundoutput device 1955 or a headphone of an external electronic device(e.g., an electronic device 1902) directly (e.g., wired) or wirelesslycoupled with the electronic device 1901.

The sensor module 1976 may detect an operational state (e.g., power ortemperature) of the electronic device 1901 or an environmental state(e.g., a state of a user) external to the electronic device 101, andthen generate an electrical signal or data value corresponding to thedetected state. According to an embodiment, the sensor module 1976 mayinclude, for example, a gesture sensor, a gyro sensor, an atmosphericpressure sensor, a magnetic sensor, an acceleration sensor, a gripsensor, a proximity sensor, a color sensor, an infrared (IR) sensor, abiometric sensor, a temperature sensor, a humidity sensor, or anilluminance sensor.

The interface 1977 may support one or more specified protocols to beused for the electronic device 1901 to be coupled with the externalelectronic device (e.g., the electronic device 1902) directly (e.g.,wired) or wirelessly. According to an embodiment, the interface 1977 mayinclude, for example, a high definition multimedia interface (HDMI), auniversal serial bus (USB) interface, a secure digital (SD) cardinterface, or an audio interface.

A connecting terminal 1978 may include a connector via which theelectronic device 1901 may be physically connected with the externalelectronic device (e.g., the electronic device 1902). According to anembodiment, the connecting terminal 1978 may include, for example, aHDMI connector, a USB connector, a SD card connector, or an audioconnector (e.g., a headphone connector).

The haptic module 1979 may convert an electrical signal into amechanical stimulus (e.g., a vibration or motion) or electrical stimuluswhich may be recognized by a user via his tactile sensation orkinesthetic sensation. According to an embodiment, the haptic module1979 may include, for example, a motor, a piezoelectric element, or anelectric stimulator.

The camera module 1980 may capture a still image or moving images.According to an embodiment, the camera module 1980 may include one ormore lenses, image sensors, ISPs, or flashes.

The power management module 1988 may manage power supplied to theelectronic device 1901. According to one embodiment, the powermanagement module 188 may be implemented as at least part of, forexample, a power management integrated circuit (PMIC).

The battery 1989 may supply power to at least one component of theelectronic device 1901. According to an embodiment, the battery 1989 mayinclude, for example, a primary cell which is not rechargeable, asecondary cell which is rechargeable, or a fuel cell.

The communication module 1990 may support establishing a direct (e.g.,wired) communication channel or a wireless communication channel betweenthe electronic device 1901 and the external electronic device (e.g., theelectronic device 1902, the electronic device 1904, or the server 1908)and performing communication via the established communication channel.The communication module 1990 may include one or more CPs that areoperable independently from the processor 1920 (e.g., the AP) andsupports a direct (e.g., wired) communication or a wirelesscommunication. According to an embodiment, the communication module 1990may include a wireless communication module 1992 (e.g., a cellularcommunication module, a short-range wireless communication module, or aGNSS communication module) or a wired communication module 1994 (e.g., alocal area network (LAN) communication module or a power linecommunication (PLC) module). A corresponding one of these communicationmodules may communicate with the external electronic device via thefirst network 1998 (e.g., a short-range communication network, such asBluetooth™, Wi-Fi direct, or infrared data association (IrDA)) or thesecond network 1999 (e.g., a long-range communication network, such as acellular network, the Internet, or a computer network (e.g., LAN or widearea network (WAN))). These various types of communication modules maybe implemented as a single component (e.g., a single chip), or may beimplemented as multi components (e.g., multi chips) separate from eachother. The wireless communication module 1992 may identify andauthenticate the electronic device 1901 in a communication network, suchas the first network 1998 or the second network 1999, using subscriberinformation (e.g., international mobile subscriber identity (IMSI))stored in the SIM 1996.

The antenna module 1997 may transmit or receive a signal or power to orfrom the outside (e.g., the external electronic device) of theelectronic device 101. According to an embodiment, the antenna module1997 may include one or more antennas, and, therefrom, at least oneantenna appropriate for a communication scheme used in the communicationnetwork, such as the first network 1998 or the second network 1999, maybe selected, for example, by the communication module 1990 (e.g., thewireless communication module 192). The signal or the power may then betransmitted or received between the communication module 1990 and theexternal electronic device via the selected at least one antenna.

At least some of the above-described components may be coupled mutuallyand communicate signals (e.g., commands or data) therebetween via aninter-peripheral communication scheme (e.g., a bus, general purposeinput and output (GPIO), serial peripheral interface (SPI), or mobileindustry processor interface (MIPI)).

According to an embodiment, commands or data may be transmitted orreceived between the electronic device 1901 and the external electronicdevice 1904 via the server 1908 coupled with the second network 1999.Each of the electronic devices 1902 and 104 may be a device of a sametype as, or a different type, from the electronic device 1901.

According to an embodiment, all or some of operations to be executed bythe electronic device 1901 may be executed at one or more of theexternal electronic devices 1902, 1904, or 1908. For example, if theelectronic device 1901 should perform a function or a serviceautomatically, or in response to a request from a user or anotherdevice, the electronic device 1901, instead of, or in addition to,executing the function or the service, may request the one or moreexternal electronic devices to perform at least part of the function orthe service. The one or more external electronic devices receiving therequest may perform the at least part of the function or the servicerequested, or an additional function or an additional service related tothe request, and transfer an outcome of the performing to the electronicdevice 1901. The electronic device 1901 may provide the outcome, with orwithout further processing of the outcome, as at least part of a replyto the request. To that end, a cloud computing, distributed computing,or client-server computing technology may be used, for example.

The electronic device according to various embodiments may be one ofvarious types of electronic devices. The electronic devices may includeat least one of, e.g., a portable communication device (e.g., asmartphone), a computer device, a portable multimedia device, a portablemedical device, a camera, a wearable device, or a home appliance.According to an embodiment of the disclosure, the electronic devices arenot limited to those described above.

It should be appreciated that various embodiments of the disclosure andthe terms used therein are not intended to limit the techniques setforth herein to particular embodiments and that various changes,equivalents, and/or replacements therefor also fall within the scope ofthe disclosure. The same or similar reference denotations may be used torefer to the same or similar elements throughout the specification andthe drawings. It is to be understood that the singular forms “a,” “an,”and “the” include plural references unless the context clearly dictatesotherwise. As used herein, the term “A or B,” “at least one of A and/orB,” “A, B, or C,” or “at least one of A, B, and/or C” may include allpossible combinations of the enumerated items. As used herein, the terms“1st” or “first” and “2nd” or “second” may modify correspondingcomponents regardless of importance and/or order and are used todistinguish a component from another without limiting the components. Itwill be understood that when an element (e.g., a first element) isreferred to as being (operatively or communicatively) “coupled with/to,”or “connected with/to” another element (e.g., a second element), it canbe coupled or connected with/to the other element directly or via athird element.

As used herein, the term “module” includes a unit configured inhardware, software, or firmware and may interchangeably be used withother terms, e.g., “logic,” “logic block,” “part,” or “circuit.” Amodule may be a single integral part or a minimum unit or part forperforming one or more functions. For example, the module may beconfigured in an application-specific integrated circuit (ASIC).

Various embodiments as set forth herein may be implemented as software(e.g., the program 1940) containing commands that are stored in amachine (e.g., computer)-readable storage medium (e.g., an internalmemory 1936) or an external memory 1938. The machine may be a devicethat may invoke a command stored in the storage medium and may beoperated as per the invoked command. The machine may include anelectronic device (e.g., the electronic device 101) according toembodiments disclosed herein. When the command is executed by aprocessor (e.g., the processor 150), the processor may perform afunction corresponding to the command on its own or using othercomponents under the control of the processor. The command may contain acode that is generated or executed by a compiler or an interpreter. Themachine-readable storage medium may be provided in the form of anon-transitory storage medium. Here, the term “non-transitory” simplymeans that the storage medium does not include a signal and is tangible,but this term does not differentiate between where data issemi-permanently stored in the storage medium and where data istemporarily stored in the storage medium.

According to an embodiment, a method according to various embodiments ofthe disclosure may be included and provided in a computer programproduct. The computer program products may be traded as commoditiesbetween sellers and buyers. The computer program product may bedistributed in the form of a machine-readable storage medium (e.g., acompact disc read only memory (CD-ROM)) or online through an applicationstore (e.g., Playstore™). When distributed online, at least part of thecomputer program product may be temporarily generated or at leasttemporarily stored in a storage medium, such as the manufacturer'sserver, a server of the application store, or a relay server.

According to various embodiments, each component (e.g., a module orprogram) may be configured of a single or multiple entities, and thevarious embodiments may exclude some of the above-described subcomponents or add other sub components. Alternatively or additionally,some components (e.g., modules or programs) may be integrated into asingle entity that may then perform the respective (pre-integration)functions of the components in the same or similar manner. According tovarious embodiments, operations performed by modules, programs, or othercomponents may be carried out sequentially, in parallel, repeatedly, orheuristically, or at least some operations may be executed in adifferent order or omitted, or other operations may be added.

As is apparent from the forgoing description, according to variousembodiments, an electronic device and a method of executing a functionof an electronic device may identify the state of an application beingexecuted by entering a designated utterance although the electronicdevice is a predetermined distance away from the user or is operated ina hold mode so that the user cannot identify the current state of theelectronic device.

According to various embodiments, an electronic device and a method ofexecuting a function of an electronic device may identify not only atask of an application being executed but also whether to performanother task executable in connection with the current task and allowthe user to control an application to operate even without an additionalutterance to operate the application.

While the disclosure has been shown and described with reference tovarious embodiments thereof, it will be understood by those skilled inthe art that various changes in form and details may be made thereinwithout departing from the spirit and scope of the disclosure as definedby the appended claims and their equivalents.

What is claimed is:
 1. An electronic device, comprising: a communicationcircuit; a microphone; a memory configured to store instructions; and atleast one processor operatively connected with the communicationcircuit, the microphone, and the memory, wherein, upon execution of thestored instructions, the at least one processor is configured to:perform a first operation, receive an utterance through the microphonewhile performing the first operation, the utterance including at leastone selected wake-up word used to invoke a speech-based intelligentservice, determine, based on the utterance, a response related to thefirst operation, and output the response.
 2. The electronic device ofclaim 1, wherein the stored instructions further configure the at leastone processor to determine the response by using a server.
 3. Theelectronic device of claim 2, wherein the stored instructions furtherconfigure the at least one processor to: perform the first operation,after receiving the utterance, transmit data related to the firstoperation to a server, receive the response from the server, and outputan aural response based on the response.
 4. The electronic device ofclaim 2, wherein the instructions further configure the at least oneprocessor to: transmit data related to the utterance to the server,request the server to identify the response related to the firstoperation, and receive the response from the server.
 5. The electronicdevice of claim 1, wherein the first operation includes at least one ofa call event, an alarm event, a timer event, a music playback event, avideo playback event, a hands-free event, or a content downloadingevent.
 6. The electronic device of claim 1, wherein the responseincludes at least one of information about the first operation, aprogress of the first operation, or an operation subsequent to the firstoperation.
 7. The electronic device of claim 6, wherein the storedinstructions further configure the at least one processor to: determinewhether progress information is changed within a designated time afterthe determining of the response, and output another response includingthe progress information.
 8. The electronic device of claim 1, whereinthe stored instructions further configure the at least one processor to:receive a user input for the response, determine a second operationrelated to the first operation when the received user input correspondsto a designated input, and further output a second response to determinewhether to perform the second operation.
 9. The electronic device ofclaim 1, wherein the stored instructions further configure the at leastone processor to: receive a user input for the response, and when thereceived user input does not correspond to a designated input, performat least one function corresponding to the first operation.
 10. Theelectronic device of claim 1, wherein the stored instructions furtherconfigure the at least one processor to, when an additional utterance isnot received within a designated time after the utterance, transmit datarelated to the first operation to a server.
 11. The electronic device ofclaim 1, wherein the stored instructions further configure the at leastone processor to: determine whether the electronic device is operatingin a designated mode, and after receiving the utterance, transmit datarelated to the first operation to the server based on a determinationthat the electronic device is operating in the designated mode.
 12. Amethod of an electronic device, the method comprising: performing afirst operation; receiving an utterance through a microphone of theelectronic device while performing the first operation; determining,based on the utterance, a response related to the first operation; andoutputting the response, wherein the utterance includes at least oneselected wake-up word used to invoke a speech-based intelligent service.13. The method of claim 12, wherein the determining of the responsecomprises using a server, and wherein the method further comprises:transmitting data related to the utterance to the server, and receiving,from the server, the response.
 14. The method of claim 12, wherein thefirst operation includes at least one of a call event, an alarm event, atimer event, a music playback event, a video playback event, ahands-free event, or a content downloading event.
 15. The method ofclaim 12, wherein the response includes information about at least oneof the first operation, a progress of the first operation, or anoperation subsequent to the first operation.
 16. The method of claim 15,further comprising: determining whether progress information is changedwithin a designated time after the determining of the response; andoutputting another response including the progress information.
 17. Themethod of claim 12, further comprising: receiving a user input for theresponse; determining a second operation related to the first operationwhen the user input corresponds to a designated input; and outputting asecond response to determine whether to perform the second operation.18. The method of claim 12, further comprising: receiving a user inputfor the response; and when the user input does not correspond to adesignated input, performing at least one function corresponding to thefirst operation.
 19. The method of claim 12, further comprising, when anadditional utterance is not received within a designated time afterreceiving the utterance, transmitting data related to the firstoperation to a server.
 20. The method of claim 12, further comprising:determining whether the electronic device is operating in a designatedmode; and after receiving the utterance, transmitting data related tothe first operation to a server based on a determination that theelectronic device is operating in the designated mode.